Attributing causality to program execution capacity modifications

ABSTRACT

Techniques are described for managing program execution capacity, such as for a group of computing nodes that are provided for executing one or more programs for a user. In some situations, dynamic program execution capacity modifications for a computing node group that is in use may be performed periodically or otherwise in a recurrent manner, such as to aggregate multiple modifications that are requested or otherwise determined to be made during a period of time. In addition, various operations may be performed to attribute causality information or other responsibility for particular program execution capacity modifications that are performed, including by attributing a single event as causing one capacity modification, and a combination of multiple events as possible causes for another capacity modification. The techniques may in some situations be used in conjunction with a fee-based program execution service that executes multiple programs on behalf of multiple users of the service.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,such as with the computing systems being co-located (e.g., as part of alocal network) or instead located in multiple distinct geographicallocations (e.g., connected via one or more private or publicintermediate networks). For example, data centers housing significantnumbers of interconnected computing systems have become commonplace,such as private data centers that are operated by and on behalf of asingle organization, and public data centers that are operated byentities as businesses to provide computing resources to customers. Somepublic data center operators provide network access, power, and secureinstallation facilities for hardware owned by various customers, whileother public data center operators provide “full service” facilitiesthat also include hardware resources made available for use by theircustomers. However, as the scale and scope of typical data centers hasincreased, the task of provisioning, administering, and managing thephysical computing resources has become increasingly complicated.

The advent of virtualization technologies for commodity hardware hasprovided some benefits with respect to managing large-scale computingresources for many users with diverse needs, allowing various computingresources to be efficiently and securely shared by multiple users. Forexample, virtualization technologies such as those provided by VMWare,XEN, Linux's KVM (“Kernel-based Virtual Machine”), or User-Mode Linuxmay allow a single physical computing machine to be shared amongmultiple users by providing each user with one or more virtual machineshosted by the single physical computing machine, with each such virtualmachine being a software simulation acting as a distinct logicalcomputing system that provides users with the illusion that they are thesole operators and administrators of a given hardware computingresource, while also providing application isolation and security amongthe various virtual machines.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are network diagrams illustrating example embodiments ofinteractions to manage program execution capacity available to multipleusers of a program execution service.

FIGS. 2A and 2B illustrate examples of managing program executioncapacity of a group of multiple computing nodes for a user, such as todynamically modify the available program execution capacity at varioustimes and in various manners.

FIG. 3 is a block diagram illustrating an example embodiment of acomputing system for managing program execution capacity provided tomultiple users.

FIG. 4 illustrates a flow diagram of an example embodiment of a ProgramExecution Service System Manager routine.

FIG. 5 illustrates a flow diagram of an example embodiment of aRecurrent Capacity Harmonization routine.

FIG. 6 illustrates a flow diagram of an example embodiment of a CapacityModification Attribution routine.

FIG. 7 illustrates a flow diagram of an example embodiment of a ProgramExecution Service Capacity Maintenance Manager routine.

DETAILED DESCRIPTION

Techniques are described for managing program execution capacity used toexecute programs for one or more users. In at least some embodiments,the program execution capacity being managed includes a group of one ormore computing nodes that are provided for use by a user in executingone or more programs. In addition, the group of computing nodesassociated with a user may be dynamically modified while in use in orderto manage an amount of program execution capacity that is available tothe user from the computing nodes of the group. The modifications to thegroup of computing nodes associated with a user may have various formsin various embodiments (e.g., to modify a quantity of computing nodes inthe group, such as by dynamically adding and/or removing computingnodes), and may be initiated in various manners in various embodiments(e.g., based on dynamic instructions specified by the user, based onautomated determinations of satisfaction of triggers that are previouslydefined by the user, based on automated operations of a service that isproviding the computing nodes of the group, etc.). Additional detailsrelated to the dynamic modification of program execution capacityavailable from a group of computing nodes are included below. Inaddition, in at least some embodiments, the techniques may be used inconjunction with a program execution service (“PES”) that executesmultiple programs on behalf of multiple customers or other users of theservice, such as a network-accessible program execution service thatprovides multiple computing nodes (e.g., multiple physical computingsystems and/or virtual machines that are hosted on one or more physicalcomputing systems) for executing programs of remote users. Some or allof the techniques may also be automatically performed by embodiments ofa Program Execution Service System Manager module and/or a ProgramExecution Service Capacity Maintenance Manager module, as described ingreater detail below.

As previously noted, the dynamic modifications that are performed to agroup of computing nodes that is in use executing one or more programsfor a user may have various forms and may be initiated in variousmanners in various embodiments. As one example, the program executioncapacity of the group of computing nodes may be measured at least inpart by the quantity of computing nodes that are part of the group, andmay be modified by changing the computing node quantity of the group(e.g., to increase the program execution capacity by increasing thecomputing node quantity, and to decrease the program execution capacityby decreasing the computing node quantity). Such computing node quantitymodifications may be used, for example, in situations in which some orall of the computing nodes in the group provide or have access to thesame or similar amounts of computing resources (e.g., amounts of memory,hard drive space, CPU execution cycles, network bandwidth, etc.), suchthat a given percentage change in computing node quantity corresponds tothe same or similar percentage change in total computing resources andprogram execution capacity of the group. In other embodiments, some orall of the computing nodes of the group may differ in one or moresignificant manners with respect to the amount of computing resources towhich they have access (e.g., if two or more distinct types of computingnode configurations are used, if each computing node is configuredindependently of the other computing nodes, etc.) or otherwise withrespect to the types of program execution capacity that they provide(e.g., based on specialized hardware, types of software programs, etc.),but dynamic program execution capacity modifications for the group maynonetheless be based at least in part on modifying a quantity of thecomputing nodes of the group, or instead may be based in other manners(e.g., changing from one type of computing node configuration toanother), as discussed in greater detail elsewhere.

Furthermore, in at least some embodiments and situations, the programexecution capacity of a group of one or more computing nodes may bemeasured and modified in manners other than a quantity of the computingnodes, such as based on an aggregate amount of one or more types ofcomputing resources provided by the group (e.g., amounts of memory, harddrive space, CPU execution cycles, network bandwidth, etc.). Inaddition, in at least some embodiments, additional factors may beconsidered in measuring and specifying program execution capacity for agroup of computing nodes, such as a geographical location of some or allof the computing nodes, inter-relationships between some or all of thecomputing nodes (e.g., to be separated by at most a maximum geographicaldistance or at least a minimum geographical distance, to be separated byat most a maximum network latency or at least a minimum network latency,to be separated into two or more independent data centers or othercomputing node collections that are unlikely to fail simultaneously,etc.), availability of specialized hardware capabilities and/or softwarecapabilities, etc. Additional details related to measuring andspecifying program execution capacity are included below.

A PES or other system that is managing the computing nodes of a groupthat is in use executing one or more programs for a user mayautomatically determine how and when to make dynamic program executioncapacity modifications for the computing node group in various manners.For example, in at least some embodiments and situations, the PES orother system may make some types of program execution capacitymodifications in an immediate manner, while other types of programexecution capacity modifications may be performed periodically orotherwise in a recurrent manner (e.g., so as to defer and aggregatemultiple modifications that are requested or otherwise determined to bemade during a period of time, such as since a prior performance of oneor more aggregated modifications). If multiple program executioncapacity modification determinations are aggregated over a period oftime, information about the aggregated modification determinations maybe used to enhance the performance of the program execution capacitymodifications in various manners. For example, if two determined programexecution capacity modifications correspond to opposite types ofmodifications (e.g., to increase and decrease computing node quantity,to increase and decrease available aggregate memory, etc.), the twomodifications may be aggregated in various manners, such as by beingselected to partially or completely offset each other, or instead byselecting a higher priority of the two modifications to be performed inlieu of the other modification. In addition, if two or more determinedprogram execution capacity modifications correspond to similar orcomplementary types of modifications (e.g., to all increase computingnode quantity by specified amounts, to all increase available aggregatememory by specified amounts, etc.), those determined modifications maysimilarly be aggregated in various manners (e.g., to select a singledetermined modification that satisfies some specified criteria, such asthe largest, the smallest, the one with the highest priority, the onethat was determined first, the one that was determined last, etc.; toaccumulate the various determined modifications and use the accumulatedmodification amount; etc.). Additional details related to determininghow and when to make various types of program execution capacitymodifications are included below.

In addition, when a PES or other system dynamically performs programexecution capacity modifications to a group of computing nodes that isexecuting one or more programs on behalf of a user, the PES or othersystem may further perform various operations to attribute causalityinformation or other responsibility for particular program executioncapacity modifications. The attribution of responsibility may include,for example, identifying particular events that have occurred during arelevant time period that each are capable of causing a dynamic programexecution capacity modification, and attributing one or more of theevents to some or all of the dynamic program execution capacitymodifications that are performed during or subsequent to that timeperiod. For example, some dynamic program execution capacitymodifications may each be initiated by a single particular event in atleast some embodiments and situations (e.g., if a computing node of thegroup fails or otherwise becomes unavailable, and the systemautomatically immediately initiates the providing of a replacementcomputing node for the group, with the computing node unavailabilitybeing a single event that directly causes the automated system actionsto provide the replacement computing node). Other dynamic programexecution capacity modifications may each be attributed to a combinationof multiple events that each contributed or may have contributed to thecapacity modification in at least some embodiments and situations (e.g.,if multiple independent events each request or indicate an increase incomputing node quantity for the group during a period of time, and areaggregated to perform a single computing node quantity increase at theend of the time period, with the various independent events beingmultiple events that together indirectly cause the later automatedsystem actions to perform the computing node quantity increase).

The attribution of responsibility for particular dynamic programexecution capacity modifications to a computing node group associatedwith a user may have particular benefits in situations such as, forexample, when the user is charged for at least some of the programexecution capacity modifications (e.g., if the PES or other system is afee-based system that charges user customers for providing eachcomputing node of the group, for each other measure of amount of programexecution capacity, and/or on other bases). In such situations, the usermay not only be able to receive and review information about chargesthat correspond to particular dynamic program execution capacitymodifications, but also the associated responsibility attributioninformation to enable the user to affirm the cause of andappropriateness of those dynamic program execution capacitymodifications. In at least some such embodiments, the responsibilityattribution information may be generated in a human-readable format fordisplay to the user, to enable the user to understand the explanation ofwhy various automated actions were taken by the PES or other system thatis included in the human-readable information. Such responsibilityattribution information may also be used in various other manners inother embodiments, including by the PES or other system to automaticallyinitiate other operations. In addition, in at least some embodiments,the responsibility attribution information may be generated and/or usedin response to various types of queries received from users or othersources, such as a request to identify which event(s) are the cause of aparticular indicated program execution capacity modification or otherchange in availability of one or more computing nodes of a group, and/orof which program execution capacity modification(s) or other computingnode group availability change(s) are caused by one or more indicatedevents. Additional details related to determining and usingresponsibility attribution information for dynamic program executioncapacity modifications are included below.

The described techniques for automatically managing dynamic programexecution capacity modifications may provide various benefits in varioussituations. For example, by aggregating multiple requested or determineddynamic program execution capacity modifications for considerationtogether, the PES or other system may be able to optimize how theaggregated modifications are performed, as well as to minimize repeatedchanges to the computing nodes that may create periods of temporaryunavailability of some or all of the computing nodes for additionalchanges. In addition, a user may predefine various types of triggersthat are based on performance characteristics of a group of computingnodes, with the program execution capacity being automatically increasedor decreased as appropriate as particular triggers are satisfied (e.g.,to reactively increase the program execution capacity of a group ofcomputing nodes to accommodate a temporary increase in the computingload for the computing node group; to proactively increase or decreasethe program execution capacity of a group of computing nodes toaccommodate an expected upcoming need for additional program executioncapacity and/or an expected upcoming lack of need for existing programexecution capacity, such as based on trends over time in particularperformance characteristics and/or historical data indicating recurringpatterns of program execution capacity use; etc.). Alternatively, a usermay desire to maintain the program execution capacity for a group ofcomputing nodes at or near a specified level (e.g., a specified desiredconstant quantity of computing nodes), and various modifications may beautomatically made to the group of computing nodes to maintain theavailable program execution capacity at that specified level (e.g., toreturn an actual computing node quantity that has deviated from thespecified desired quantity back to that specified desired quantity).Such techniques may be of use, for example, when each of the computingnodes of the group executes a distinct copy of the same program (e.g.,to serve as alternatives for an aggregate computing load over thegroup), and the quantity of computing nodes are modified to manage theamount of work handled by each computing node, or otherwise insituations in which the various computing nodes of a group do not eachexecute a distinct copy of the same program (e.g., if distinct subsetsof the group computing nodes each execute copies of different programs,such as to have some computing nodes execute an application serverprogram and to have other computing nodes execute an associated databaseserver program; if some or all of the computing nodes perform differentparts of a single program, such as in a distributed manner; etc.).Furthermore, when adding additional computing nodes to a group, the PESor other system may further optionally take other actions in at leastsome situations, such as to provision the added computing node to beready to execute one or more programs, or to further automaticallyinitiate the execution of the one or more programs on the addedcomputing node.

As previously noted, users may predefine various types of triggersrelated to dynamically modifying program execution capacity for groupsof computing nodes, and those triggers may be later used to initiatecorresponding automated dynamic program execution capacity modificationsfor computing node groups of the users. As one example, a trigger may bedefined that specifies a particular desired quantity of computing nodesor desired aggregate amount of one or more computing resources, such asmay be used to automatically maintain a corresponding computing nodegroup at that desired computing node quantity or desired aggregatecomputing resource amount, or instead to change to a specified computingnode quantity or aggregate computing resource amount if specifiedcriteria are satisfied. In other situations, a trigger may be definedthat specifies a particular absolute or relative change in computingnode quantity or aggregate amount of one or more computing resources,such as may be triggered if one or more indicated performancecharacteristics of the computing node group reach a specified thresholdor otherwise satisfy a specified criteria (e.g., remain within aspecified range for a specified period of time; show a particular trend,such as a specified amount of change over a specified period of time, orsuch as a particular “acceleration” or rate of change; match or othercorrespond to one or more specified patterns, such as from historicaldata that indicates recurring patterns of program execution capacityuse; satisfy a specified logical or other combination of values formultiple performance characteristics, such as may be combined usinglogical operators such as AND, NOT, OR, etc. and/or in other manners;etc.). In other embodiments and situations, a predefined trigger may beotherwise satisfied based on information that is not part of theperformance characteristics of the computing node group (e.g., based ona current time matching one or more specified times that are part or allof the criteria for a particular trigger; based on status informationfor one or more programs being executed on the current computing nodegroup that is measured in manners other than performancecharacteristics, such as a current computing load as indicated by, forexample, an amount of work that is queued for or otherwise known orexpected to be performed by the one or more programs; based onperformance characteristics or other status information for one or moreother executing programs that are not being executed by the currentcomputing node group, such as if the current computing node group isinteracting with or otherwise supporting the other executing programs soas to, for example, increase the program execution capacity of thecurrent computing node group as the computing load on the otherexecuting programs increases or decreases; etc.). Such performancecharacteristics may be based on any measurable attribute of or othermetric corresponding to the operation of one or more of the computingnodes of the group, including the following non-exclusive list: anabsolute or relative amount of use of one or more computing resources ofa single computing node (e.g., a percentage amount of available memorybeing used or CPU cycle utilization, an absolute amount of networkbandwidth being used, etc.); an absolute or relative amount of aggregateuse of one or more computing resources of all of the group computingnodes; an absolute or relative amount of latency or other delay inresponding to communications from external computing systems; anabsolute or relative amount of failures of the computing nodes inperforming one or more desired actions; etc. Furthermore, in at leastsome embodiments and situations, rather than having a trigger directlyspecify a particular program execution capacity modification that is tooccur when it is satisfied, the satisfaction of the trigger may cause adesignated system or module to be notified, and that system or modulemay request a particular program execution capacity modification (e.g.,a predefined capacity modification that does not vary; a capacitymodification that is dynamically determined at the time of thenotification, such as based on then-current conditions; etc.). Inaddition, the PES or other system may in some embodiments performvarious operations to monitor a group of computing nodes in order todetermine some or all of the performance characteristics for thetriggers associated with the group, or may otherwise obtain suchmonitored performance characteristics from another source (e.g., fromthird-party software that monitors the computing nodes, from softwareexecuting on a computing node to monitor that computing node andoptionally report the monitored information, etc.). Furthermore, in someembodiments, the PES or other system may have system-defined triggersthat initiate dynamic program execution capacity modifications whenindicated trigger criteria are satisfied, or may otherwise automaticallydetermine to make some types of changes to computing node groups inspecified circumstances.

When a defined trigger specifies a particular absolute or relativechange in computing node quantity or aggregate amount of one or morecomputing resources, and the one or more specified criteria for thedefined trigger are automatically determined to be satisfied based oncurrent conditions for a corresponding group of computing nodes, the PESor other system may automatically determine whether and how to performthe specified program execution capacity modification for the trigger.For example, some types of specified program execution capacitymodifications may be performed immediately (e.g., a request to terminateuse of one or more computing nodes, a request based on a trigger thatthe user has designated for immediate performance, etc.), while othertypes of specified program execution capacity modifications may bedeferred until the end of an aggregation period of time forconsideration as part of an aggregation of multiple program executioncapacity modifications that are requested or otherwise determined duringthat period of time. Similarly, program execution capacity modificationsthat are dynamically requested by a user (e.g., via a GUI, or graphicaluser interface of the PES or other system; by a program of the user viaa defined API, or application programming interface, of the PES or othersystem; etc.) may be determined to be performed immediately and/ortemporarily deferred and aggregated in a similar manner, such as basedon the type of dynamic program execution capacity modification, based onan explicit request from the user for immediate or deferred performance,etc. Moreover, when determining how to manage a combination of multipledetermined program execution capacity modifications, in some situationsdifferent priorities may be associated with different determinedmodifications. If so, such priorities may be assessed in variousmanners, such as for dynamically specified user requests or other userinstructions to be given a different priority from satisfied triggers(e.g., a higher or lower priority), for different types of determinedmodifications to be given different priorities (e.g., for requests todecrease program execution capacity being given higher priority thanrequests to increase program execution capacity), etc. Additionaldetails related to using user-defined triggers and determiningperformance characteristics are included below.

Furthermore, in at least some embodiments, the PES or other system maymanage periodic or otherwise recurrent aggregations of multipledetermined program execution capacity modifications based on trackingand using multiple attributes for a corresponding group of computingnodes, or otherwise with respect to controllable parameters that areeach related to one or more computing nodes of the group. As oneexample, using computing node quantity for a computing node group as ametric corresponding to program execution capacity for the computingnode group, the PES or other system may maintain and use at least threeinter-related measures for the computing node group, as follows: adesired computing node quantity for the computing node group, such asmay be initially set by an associated user when the computing node groupis initiated, and may be modified by satisfaction of triggers and/ordynamically specified user requests; an actual computing node quantityof the currently available computing nodes of the group (e.g., asdetermined by continuous or repeated monitoring of the computing nodesof the group); and an official recorded computing node quantity of thecurrently available computing nodes of the group (e.g., as determined atthe last time that one or more dynamic program execution capacitymodifications are initiated, and optionally as updated occasionally bythe continuous or repeated monitoring). Such multiple attributes for thecomputing node group may be used in various manners, such as tocontinuously or repeatedly measure the actual quantity and update theofficial recorded quantity accordingly (e.g., based on performingmonitoring of the computing nodes of the group), and to periodicallyattempt to update the most recent official recorded quantity to matchthe current desired quantity (e.g., when considering how and whether toperform a combination of multiple aggregated determined modifications).

As previously noted, the PES or other system may further perform variousoperations to attribute causality information or other responsibilityfor particular dynamic program execution capacity modifications that aremade. For example, as previously noted, events corresponding to requestsfor dynamic program execution capacity modifications for a computingnode group may be tracked, including dynamic user-specified instructionsthat are received and predefined triggers that are automaticallydetermined to be satisfied, as well as actions that are automaticallytaken by the PES or other system in some embodiments (e.g., based onmonitoring that is performed, such as if a computing node is determinedto have become frozen or otherwise unavailable to perform desiredactivities, to automatically shutdown or otherwise terminate the use ofthe computing node as part of its current computing node group).Similarly, actual changes in program execution capacity for thecomputing node group may similarly be tracked, such as changescorresponding to events and/or to other capacity changes that occur(e.g., instances of computing nodes failing or otherwise becomingunavailable). As one example, various event-related information may bestored in a first database table, and various information related tocapacity change or other availability change may be stored in a seconddatabase table. If a relationship between a particular event and aparticular availability change is identified (e.g., a particular eventcauses a particular availability change to be immediately performed), acorresponding nexus between that event and availability change may betracked by storing the same nexus-related identifier along with theother information for that event and that availability change. In othersituations in which multiple events may individually or in combinationcause a particular availability change but in which causality is notattributed to a single event, a single nexus-related identifier may bestored along with the other information for that availability change andeach of those events, and further for one or more other availabilitychanges if they similarly may be attributed to those same multipleevents individually or in combination. Thus, for example, if multipleevents for a given computing node group occur during a singleaggregation time period and one or more program execution capacitychanges occur during that same aggregation time period or immediatelyafter (e.g., as part of harmonization activities that are performed atthe end of the aggregation time period), and if none of those events aredirectly attributable to any of the one or more capacity changes, thecombination of all of the multiple events may be attributed to each ofthe one or more capacity changes. Further details related to suchcausality attribution are included below, including with respect to FIG.2B.

In addition, in a similar manner, multiple such desired, actual andofficial attributes may be tracked and used for one or more othercontrollable parameters corresponding to program execution capacity fora computing node group, such as a first set of desired, actual andofficial amounts of aggregate average CPU cycle utilization, a secondset of desired, actual and official amounts of aggregate average networkbandwidth utilization, etc. Furthermore, when multiple parameterscorresponding to program execution capacity for a computing node groupare simultaneously tracked and used, the PES or other system may attemptto manage all of the parameters, such as to modify a computing nodegroup in order to simultaneously achieve desired aggregate average CPUcycle utilization and desired aggregate average network bandwidthutilization. As another example, multiple parameters for a computingnode group may include both a quantity of computing nodes and specifiedgeographical locations of various computing nodes of the group (e.g.,between fifteen and twenty percent of the group computing nodes at afirst data center, and the remaining group computing nodes at one ormore other data centers), with the PES or other system attempting tomanage both computing node quantity and computing node geographicallocation for the group simultaneously. Additional details are includedbelow related to using multiple attributes to track and manage one ormore program execution capacity parameters.

In addition, a PES or other system may provide users with access tocomputing nodes in various manners in various embodiments. For example,in some embodiments, at least some of the computing nodes available froma PES for executing programs may be allocated to one or more users forpreferential use by those users, such that each of those users haspriority access relative to other users to use those computing nodes. Inone such embodiment, the priority access of the users may be based oneach of the users having dedicated or exclusive access to use thosecomputing nodes for a specified period of time, such as in a manneranalogous to a lease. In addition, in some embodiments, at least some ofthe computing nodes that are allocated to one or more users fordedicated or other preferential use may be used as excess programexecution capacity for other users at times, such as when the computingnodes are not in use by the users to whom the computing nodes areallocated and/or when a user to whom a computing node is allocatedexplicitly makes the allocated computing node available for use by otherusers. In this manner, at least some program execution capacity that isallocated to a first group of users may become available from time totime to temporarily execute programs on behalf of other users, such ason a non-guaranteed basis (e.g., such that access to the excess programexecution capacity may be rescinded if that program execution capacityis desired for other purposes, such as preferential or reserved use byone or more other users). Furthermore, in some embodiments, a PES mayinclude on-demand computing nodes that are available to satisfydynamically received requests of users to execute programs (e.g.,immediately upon request of those users, at an indicated future time, atsome time during an indicated future time period, etc.), such that theone or more programs indicated by such a request may be executed ifcomputing nodes sufficient to satisfy the requested execution areavailable at (or near) the requested time, but without such a requestbeing guaranteed to be satisfied. In addition, in some embodiments,after such an on-demand request for immediate (or scheduled) executionis satisfied and successfully initiates execution of one or moreprograms on behalf of a respective user, the ongoing use of theon-demand computing nodes may be guaranteed to continue until somefuture time, such as a time of the respective user's choosing,optionally subject to certain limitations (e.g., such as to beguaranteed that the PES will not preempt the use for other purposes, butnot to be guaranteed against failure of the computing nodes executingthe programs). In some embodiments, the computing nodes used to providethe on-demand variable program execution capacity may be distinct fromthe computing nodes used to provide dedicated program execution capacityand/or from the computing nodes used to provide excess program executioncapacity—thus, if some of the computing nodes used to provide theon-demand variable program execution capacity are not in use, in someembodiments they may be used to provide excess program executioncapacity until on-demand variable program execution capacity requestsare received, while in other embodiments they may not be used to provideexcess program execution capacity. In other embodiments, only a singletype of program execution capacity may be provided, and/or other typesof program execution capacity may be provided.

FIG. 1A is a network diagram that illustrates an example of a programexecution service that manages computing nodes that are available foruse in providing program execution capacity to execute programs formultiple users. For illustrative purposes, some examples and embodimentsare described below in which specific types of program executioncapability are provided and managed in specific manners. In addition, insome of the examples and embodiments described below, the programexecution capacity that is provided by a group of computing nodes may bemeasured in particular manners (e.g., based on a quantity of thecomputing nodes), may be managed in particular manners (e.g., bytracking use of desired, actual and official attributes with respect toone or more program execution capacity metrics), may be controlled byassociated users in various manners (e.g., based on the use ofpredefined triggers and/or dynamically specified instructions), may bemodified in particular manners (e.g., by aggregating at least someprogram execution capacity modifications that are determined anddeferred during a period of time and then performing one or morecorresponding aggregated modifications at the end of the time period,such as to modify the quantity of computing nodes in a group), etc.These examples are provided for illustrative purposes and are simplifiedfor the sake of brevity, and it will be appreciated that the inventivetechniques may be used in a wide variety of other situations, only someof which are described below.

In the example of FIG. 1A, various users (not shown) are using variousclient computing systems 130 to interact over a network 100 with a PESprovided by a program execution service provider entity 105, with someof the functionality of the PES being provided in this example by anillustrated embodiment of a Program Execution Service System Manager(“PESSM”) module 110, and other functionality being provided in thisexample by an illustrated embodiment of a Program Execution ServiceCapacity Maintenance Manager (“PESCMM”) module 115. The PESSM module 110may, for example, assist particular users in configuring groups ofcomputing nodes to be used to execute programs for the users, includingspecifying initial desired computing node quantities for the groups andspecifying triggers for use in later automatically making dynamicmodifications to the computing node quantities. In this example, the PESmakes various computing nodes 120 available for executing programs ofthe users, although in other embodiments at least some of the computingnodes used for at least some of the groups may be provided in othermanners (e.g., made available by the users and/or by third-parties, suchas external computing systems 140, but managed by the PES). In addition,the PESCMM module 115 may assist in continuously or repeatedlymonitoring computing node groups that are in use, and optionallyattempting to replace any computing nodes that fail or otherwise becomeunavailable, so as to maintain program execution capacity at previouslydetermined levels.

The network 100 may, for example, be a publicly accessible network oflinked networks, possibly operated by various distinct parties, such asthe Internet. In other embodiments, the network 100 may be a privatenetwork, such as, for example, a corporate or university network that iswholly or partially inaccessible to non-privileged users. In still otherembodiments, the network 100 may include one or more private networkswith access to and/or from the Internet. In the illustrated embodiment,the PESSM module 110 and PESCMM module 115 may each include softwareinstructions that execute on one or more computing systems (not shown).In addition, the modules 110 and 115 and various computing nodes 120 maybe provided in various manners, such as at a single data center orotherwise to use a group of co-located computing systems, or instead ina distributed manner using various computing systems in various distinctgeographical locations.

In some embodiments, the illustrated computing nodes 120 may includemultiple physical computing systems and/or multiple virtual machinesthat are hosted on one or more physical computing systems (as isdescribed in more detail with respect to FIG. 1B). Each of the computingnodes 120 has some amount of computing resources available for executingone or more programs, such as to provide a specific amount of programexecution capacity, such as may be measured, for example, by acombination of one or more of processing capacity (e.g., number and/orsize of processing units), memory capacity, storage capacity, networkbandwidth capacity, etc. In some embodiments, the PES provider 105 mayprovide preconfigured computing nodes, with each preconfigured computingnode having equivalent or otherwise similar amounts of resourcesavailable for executing programs on behalf of users, while in otherembodiments, the PES provider 105 may provide a selection of variousdifferent computing nodes from which a user may choose for executingprograms on behalf of the user, such as with each selection havingvarying amounts and/or types of computing resources (e.g., size, speedand/or type of processing units; number of processing units; amount ofmemory and/or storage; platform configuration, such as 32-bit or 64-bit;etc.).

In this illustrated embodiment, the program execution service providesfunctionality for managing groups of one or more computing nodes 120 foreach of multiple users. As discussed in greater detail elsewhere, thevarious users may interact with the PESSM module 110 to specify requeststo initiate use of groups of computing nodes for execution of programson behalf of the users. In various embodiments, such resources may bespecified at the time of a request for execution of one or more programson a group of computing nodes on behalf of a user and/or at one or moreother times, such as when a user initially registers and/or subscribesto use services of the PES. In some embodiments, the PESSM module 110may provide subscription and/or registration services to one or moreusers, such that users may specify information related to one or moreprograms to execute on behalf of a user (e.g., programs, source code,addressable locations of one or more programs, etc.), accountinformation (e.g., user name, billing information, etc.), terms of use,etc. In some embodiments, after a user interacts with the PESSM module110 to subscribe and/or register for services, the user may be issuedone or more identifiers (e.g., keys, tokens, user names, etc.) that areassociated with the user and are to be used in conjunction withexecuting programs on behalf of the user. In other embodiments, a moduleother than the PESSM module 110, not shown, may be provided to performvarious operations related to subscription and/or registration servicesof a PES.

After a request is received from a user for use of one or more computingnodes, the PESSM 1 module 10 may determine whether there are asufficient number of computing nodes 120 with available resources forsatisfying the request, and if so, the PESSM 1 module 10 may initiateexecution of one or more programs for the request on an appropriateamount of the computing nodes on behalf of the user. In cases where auser schedules a request for future execution of one or more programs ona group of one or more computing nodes, the PESSM module 110 may attemptto immediately reserve an appropriate amount of computing nodes forexecuting the one or more programs at the one or more future times,and/or may delay the determination of which computing nodes to use forexecution until a later time (e.g., such as when the one or more futuretimes occur). In the illustrated embodiment, if the PESSM module 110 isunable to allocate computing nodes for a user request, the request mayfail, such that the programs are not executed. In such cases, the usermay resubmit a failed request for later execution. As previously noted,in some embodiments, a user may be charged various fees in associationwith use of the PES, such as based on a number of computing nodes used,a type of computing nodes used, a duration of time the computing nodesare used, particular operations that the computing nodes perform (e.g.,data transfer and/or storage), etc.

After a group of one or more computing nodes is provided for use inexecuting one or more programs on behalf of a user, the computing nodegroup may be managed in various manners. For example, as previouslynoted, the PESCMM module 115 may monitor the computing nodes of thegroup, such as to determine performance characteristics of some or allof the computing nodes, including an actual computing node quantity orother measure of actual program execution capacity being provided by thecomputing nodes of the group. The actual program execution capacity ofthe group may change if, for example, one or more of the computing nodesfail or otherwise become unavailable, and in at least some embodiments,the module 115 may perform various operations to maintain the programexecution capacity of the computing node group in such situations (e.g.,by initiating the addition of replacement computing nodes to the groupin place of the unavailable computing nodes). In addition, the module115 may use information about a determined actual computing nodequantity of the computing node group to update an official recordedcomputing node quantity of the computing node group, such as upondetecting a change in the actual computing node quantity, periodically,etc.

In addition, the PESSM module 110 may also assist in managing thecomputing node group in various manners. For example, as previouslynoted, the user associated with the group may have previously specifiedone or more quantity modification triggers that specify types ofcomputing node quantity changes to be made to the group if variousspecified criteria are satisfied. Furthermore, in at least someembodiments, the user associated with the group may dynamically specifychanges to the operation of their associated computing node group atvarious times, including to change the quantity of the computing nodesof the group. As part of managing the computing node group, the module110 may track the desired computing node quantity for the computing nodegroup, and periodically harmonize the official recorded computing nodequantity for the computing node group with the desired computing nodequantity for the computing node group. Such periodic harmonization mayinclude tracking and aggregating requested modifications to thecomputing node quantity that occur during an aggregation period of time,such as from satisfaction of one or more of the predefineduser-specified triggers and/or from one or more dynamically specifieduser instructions. At the end of the time period, the module 110 maythen determine whether and how to perform the aggregated computing nodequantity modifications, and may update the desired computing nodequantity based on the aggregation of the computing node quantitymodifications. The module 110 may then complete the harmonization of thedesired and official recorded computing node quantities by initiatingthe aggregated computing node quantity modifications, and updating theofficial recorded computing node quantity to match the new desiredcomputing node quantity that will result from the aggregated computingnode quantity modifications. Furthermore, in cases in which the currentactual computing node quantity differs from the official recordedcomputing node quantity at the end of the time period, the harmonizationof the desired and official recorded computing node quantities mayfurther be performed in light of the actual computing node quantity,such that the initiation of the aggregated computing node quantitymodifications is performed to update the current actual computing nodequantity to the current desired computing node quantity. The module 110may further perform such periodic harmonization activities repeatedly,such as each time a specified period of time has elapsed, each time aminimum amount of computing node quantity changes have been requested,etc.

Furthermore, the PESSM module 110 may perform other activities to trackevents and particular program execution capacity changes that occur withrespect to particular computing node groups. At least some of theprogram execution capacity change information may correspond to dynamicprogram execution capacity modifications that are initiated by themodule 110 as part of periodic or recurrent harmonization activities,and optionally in some situations may further include program executioncapacity change information from the monitoring performed by the PESCMMmodule 115 (e.g., to correspond to computing nodes that fail orotherwise become unavailable, or other types of detected changes inprogram execution capacity for computing node groups). The module 110may further make automated determinations at various times to attributecausality or other responsibility for particular capacity changes toparticular events, such as in accordance with periodic or otherrecurrent harmonization activities that are performed, or instead atother times.

Although the foregoing example embodiment of FIG. 1A is described withrespect to a PES that provides various types of functionality forvarious users, it will be appreciated that various other embodiments mayexist. For example, in at least some embodiments, unused portions of asingle one of the computing nodes 120 (e.g., unused processing unitclock cycles, unused portions of memory, etc.) may be made available foruse by one or more users, such that one or more programs of a first usermay be sharing resources of a single computing node with those of one ormore other users. In addition, although some embodiments are describedwith respect to a program execution service and program executioncapacity, it will be appreciated that the described techniques may beused to manage access to various other groups of computing nodes orother types of computing-related resources. A non-exclusive list ofexamples of other types of computing-related resources that may bemanaged for use by multiple users may include the following: persistentdata storage capabilities (e.g., on non-volatile memory devices, such ashard disk drives); temporary data storage capabilities (e.g., onvolatile memory, such as RAM); message queuing and/or passingcapabilities; other types of communication capabilities (e.g., networksockets, virtual communication circuits, etc.); database managementcapabilities; dedicated bandwidth or other network-related resources;input device capabilities; output device capabilities; CPU cycles orother instruction execution capabilities; etc.

FIG. 1B illustrates an embodiment in which a program execution servicemay be provided using one or more data centers that include multiplephysical computing systems. In particular, FIG. 1B is a network diagramillustrating an example embodiment in which a PESSM module 180 andPESCMM module 160 of a program execution service manage execution of oneor more programs on behalf of users using various computing systems atthe one or more data centers. The illustrated example includes a datacenter 170 used by the PES, which is connected to the Internet 196external to the data center 170. In this example, the Internet 196provides access to various external computing systems, such as computingsystems 190 via private network 194 and directly accessible computingsystems 192. The private network 194 may be, for example, a corporatenetwork that is wholly or partially inaccessible from non-privilegedcomputing systems external to the private network 194. Computing systems192 may include, for example, a home computing system that connectsdirectly to the Internet (e.g., via a telephone or cable modem, aDigital Subscriber Line (“DSL”), etc.). In addition, one or more otherdata centers 198 are illustrated that are connected to data center 170via the Internet 196, such as may further be used by the PES in at leastsome embodiments.

The example data center 170 includes a number of physical host computingsystems 175, physical computing systems 182, a PESSM module 180 of thePES, and a PESCMM module 160 of the PES. In this example, host computingsystems 175 each provide multiple virtual machines and have a virtualmachine (“VM”) manager component to manage those virtual machines (e.g.,a hypervisor or other virtual machine monitor), such as is illustratedwith respect to host computing system 175 a with multiple virtualmachines 177 a and a VM manager component 179 a. The other hostcomputing systems 175 b may similarly include such components, but thoseother components are not illustrated here for the sake of brevity, andsome or all of the computing systems 182 may optionally similarly haveone or more such virtual machines and/or VM manager components (notshown). Each of the virtual machines provided by a host computing systemmay be used as a distinct computing node for the PES, such as to have afirst virtual machine computing node on a host computing system be partof a first computing node group for a first user, and to have a secondvirtual machine computing node on that same host computing system bepart of a second computing node group for a second user. Alternatively,in other embodiments, some or all of the physical host computing systemsat the data center may not provide any virtual machines, such as toinstead directly act as a computing node that executes one or moreprograms on behalf of an end user customer of the PES. Furthermore, insome embodiments various of the computing systems 175 and 182 may havediffering capabilities, may have different associated fees for use, maysupport different types of user programs (e.g., virtual machine softwareimage instances of different sizes, or programs with different types ofresource criteria and/or computing resource usage, such as differingpatterns of I/O and memory access and network usage), etc. If so,particular users and/or their programs may be grouped (e.g.,automatically) according to one or more such factors, which may furtherbe used as constraints and/or preferences regarding which computingsystems to select for executing particular programs. The example datacenter 170 further includes an internal network 172 that may includemultiple networking devices (not shown), such as switches, edge routers,and core routers, with computing systems 175 and 182, the PESCMM module160, and the PESSM module 180 connected to the internal network 172. Thevarious host computing systems 175 and other computing systems 182 maybe arranged in various manners, including by being grouped in racks thatshare common backplanes or other interconnection mediums. Furthermore,each of the modules 160 and 180 may be executed using one or morecomputing systems (not shown).

The illustrated PESSM module 180 and PESCMM module 160 perform at leastsome of the described techniques in order to manage execution ofprograms on groups of computing nodes that are provided using thecomputing systems 175 and 182, as described in greater detail elsewhere.When a particular computing node is selected to execute one or moreprograms of a user, the PESSM module may in some embodiments initiateexecution of those programs by interacting with a VM manager componentor other manager component that controls execution of programs for thatselected computing node, or may alternatively directly execute theprograms on the selected computing node. Users of the PES may usevarious computing systems to interact with the PESSM module 180, such ascomputing systems 190 or 192, or computing systems at one of the otherdata centers 198.

It will be appreciated that the data center of FIG. 1B is provided forillustrative purposes only, and that program execution services andother software execution services may be provided in other manners inother embodiments. For example, PESSM module 180 and/or PESCMM module160 may instead be provided using one or more other computing systemsexternal to the data center 170, such as computing systems 190, 192 orat a data center 198.

FIG. 2A illustrates an example of techniques for managing an examplegroup of computing nodes that are provided to execute one or moreprograms of an example user, such as techniques that may beautomatically performed by embodiments of a PESSM module and/or a PESCMMmodule. In particular, in this example, a particular user (referred toas User UUU below, and not shown in FIG. 2A) has initiated the use of agroup of multiple computing nodes to each execute a copy of an indicatedprogram on behalf of the user, such as to serve as alternative computingnodes to handle requests received by a service that is provided by theexecuting program (e.g., in order to balance the computing load for theservice across the multiple computing nodes of the group). As isillustrated in the timeline graph 210, the user has requested at time T1that the computing node group be provided, and has specified an initialdesired computing node quantity 215 a of 8 computing nodes for thegroup. Information 250 indicates various user-defined triggers that theuser has specified for the computing node group, such as at the time ofthe initial request. In addition, timeline graph 205 illustratesinformation about two example types of performance characteristics thatwill be tracked for the computing node group and that will be used todetermine whether the triggers 250 are satisfied, which in this exampleincludes aggregate average CPU utilization 205 a for the computing nodegroup and aggregate average network bandwidth utilization 205 b for thecomputing node group.

In response to the event E1 at time T1 corresponding to the receiveduser request to initiate the providing of the computing node group, anexample PES (not shown) initiates a computing node group for the userthat initially includes eight computing nodes, in accordance with theinitial desired computing node quantity. In addition, an officialrecorded computing node quantity 225 for the computing node group issimilarly set to eight computing nodes. As is shown in the timelinegraph 210, however, the eight computing nodes are not actuallyimmediately available, as it takes some time to provision and makeavailable the computing nodes for use as part of the computing nodegroup, including having the program copy be executed on each of thecomputing nodes by the PES or the user. In particular, after an initialtime has passed after time T1, a change C1 255 occurs in the programexecution capacity for the computing node group, which is completed atapproximately time T2 after having been begun at time T1, and whichcorresponds to a first four of the eight computing nodes beingavailable. Accordingly, the actual computing node quantity 220 that istracked for the computing node group increases from 0 to 4 at that time.In addition, at the same time or shortly thereafter, the timeline graph205 indicates that the aggregate average CPU utilization 205 a andaggregate average network bandwidth utilization 205 b performancecharacteristics begin to be tracked based on operation of the availablecomputing nodes of the group.

In this example, the PES (or other third-party system) performsmonitoring of the computing nodes of the group in a substantiallycontinuous manner, such that the performance characteristics 205 a and205 b and the actual computing node quantity 220 information ismaintained in an up-to-date manner. However, in this example, at leastsome types of dynamic modifications to the computing node quantity forthe computing node group are performed only periodically, such as toaggregate at least some types of requested modifications duringaggregation periods of time 260, and to perform harmonization activitiesat the end of the aggregation time periods between the desired, actual,and official recorded computing node quantities 215, 220 and 225,respectively. Accordingly, during a first aggregation time period 260 a,additional changes 255 in the computing node quantity of the computingnode group occur. For example, after the change C1 that makes the firstfour computing nodes available, a subsequent change C2 that is completedat approximately time T3 corresponds to three additional computing nodesbecoming available for use as part of the computing node group.

However, in this example, the eighth computing node that was initiallyrequested does not become available in a timely manner, such as due to ahardware problem with the computing node that is initially selected (orinstead in other situations due to only 7 computing nodes beingavailable at time T1 for use in the computing node group, with theeighth computing node to be dynamically added as soon as it laterbecomes available). Accordingly, an event E2 265 is shown occurringshortly after the change C2, at approximately time T4, in which the PESterminates the original eighth computing node if needed (e.g., if it isfrozen in an intermediate state), and initiates the providing of areplacement eighth computing node. As discussed in greater detail withrespect to FIG. 2B, in this example, the PES had initiated a change C10a at time T1 that corresponds to adding the initial eighth computingnode, but that change C10 a fails to be completed with an actual eighthcomputing node that is added and becomes available, and thus the changeC10 a is not reflected in the actual computing node quantity 220 oftimeline graph 210, nor otherwise shown in FIG. 2A. Instead, in thisexample, the event E2 is separated into two distinct events E2 a and E2b (not shown separately) that occur at or near the same time. Inparticular, the PES records an event E2 a at time T4 in this examplethat corresponds to the eighth computing node failing to initializecorrectly (e.g., within a specified deadline), and the PES automaticallyinitiates a change C10 b (not shown) at time T4 to terminate the initialeighth computing node that failed to initialize correctly. Furthermore,in this example, the actual termination of the initial eighth computingnode that occurs as the result of event E2 a is itself treated as aseparate event E2 b at time T4 that automatically initiates theimmediate providing of the replacement eighth computing node. Suchchaining of events and results of corresponding changes, such that theresult from the change for a first event may itself be treated as asecond event that causes another change, may provide various benefits intracking inter-relationships between events and corresponding changeresults, as discussed in greater detail elsewhere. Subsequently, achange C3 255, which was initiated at time T4 based on event E2 b and iscompleted at time T5, brings the actual computing node quantity fromseven to eight, as the replacement eighth computing node becomesavailable for use as part of the computing node group.

After the eight computing nodes of the computing node group operate inthe intended manner for a portion of the aggregation time period 260 a,another change C4 255 occurs in the actual computing node quantity 220at approximately time T7, prompted by one of the computing nodes failingor otherwise becoming unavailable. A corresponding event E3 265 occursat approximately the same time as the change C4, in which the PESterminates the unavailable computing node as needed, and optionallyautomatically initiates the providing of a replacement computing node.In particular, in a manner similar to that previously discussed withrespect to event E2, the event E3 is separated in this example into twodistinct events E3 a and E3 b (not shown separately) that occur at ornear the same time. Thus, the PES records an event E3 a at time T7 inthis example that corresponds to the computing node being detected tohave become unavailable, and the PES automatically initiates activitiesat time T7 to terminate the unavailable computing node, such as todirectly cause the change C4. Furthermore, in this example, thetermination of the unavailable computing node that occurs as the resultof event E3 a is itself treated as a separate event E3 b thatautomatically initiates at approximately time T7 the immediate providingof a replacement computing node, although in other embodiments andsituations, any such providing of a replacement computing node willinstead be deferred until the harmonization activities H1 that areperformed at the end of the aggregation time period 260 a. Subsequently,in this example, a change C5 255 is completed at approximately time T9that returns the actual computing node quantity to eight, directlycaused by the event E3 b, as the replacement computing node becomesavailable for use as part of the computing node group. In this example,the official recorded computing node quantity 225 is not temporarilyupdated to correspond to changes C4 and C5, nor to the previous changesC1-C3, although in other embodiments the official recorded computingnode quantity 225 may be updated to reflect some or all such changes,such as to continuously or repeatedly maintain the official recordedcomputing node quantity 225 in a state that is reconciled with theupdated actual computing node quantity 220. No other computing nodecapacity availability changes or events occur during the remainder ofthe aggregation time period 260 a in this example, including none of thespecified triggers 250 being satisfied, and no dynamically specifieduser instructions being received. Accordingly, at time T11 at the end ofthat time period 260 a, a first set H1 of harmonization activities areconsidered, but no activities are needed, since the current desired,actual and official recorded computing node quantities 215, 220 and 225,respectively, are all matching at eight computing nodes. If the changeC5 to replace the unavailable computing node had not been performedduring the aggregation time period 260 a as illustrated in this example,it would instead be initiated as part of the harmonization activities H1after the aggregation time period in order to replace the unavailablecomputing node at that time (e.g., in conjunction with any other dynamicavailability changes that are requested or initiated during theaggregation time period 260 a).

During the second aggregation time period 260 b, additional events dooccur, however. In particular, an event E5 265 first occurs atapproximately time T16, corresponding to an automated determination bythe PES that trigger TR-1 250 a has been satisfied by the increasingaggregate average CPU utilization 205 a, as is graphically illustratedin the timeline graph 205 with a first black oval on the aggregateaverage CPU utilization 205 a line. The trigger TR-1 satisfactioninitiates a request to increase the desired computing node quantity 215by 4 computing nodes, for a total requested desired computing nodequantity 215 c of twelve (or alternatively could request that theupdated desired computing node quantity be set to twelve, regardless ofthe current desired computing node quantity). Similarly, after furtheradditional time has passed, an event E6 265 next occurs at approximatelytime T18, corresponding to a determination that trigger TR-N 250 c hasbeen satisfied by the increasing aggregate average network bandwidthutilization 205 b, as is graphically illustrated in the timeline graph205 with a black oval on the aggregate average network bandwidthutilization 205 b line. The trigger TR-N satisfaction initiates arequest to increase the desired computing node quantity 215 by 2computing nodes, for a total requested desired computing node quantity215 d of ten (or alternatively could request that the updated desiredcomputing node quantity be set to ten, regardless of the current desiredcomputing node quantity). After a short additional time has passed, anevent E4 265 occurs at approximately time T19, in which the userassociated with the computing node group provides a dynamicallyspecified request to increase the desired computing node quantity 215 by3 computing nodes, for a total requested desired computing node quantity215 b of eleven (or alternatively could request that the updated desiredcomputing node quantity be set to eleven, regardless of the currentdesired computing node quantity). This request may be made, for example,based on the user noticing that the aggregate average CPU utilization205 a is high, that the total computing load on the computing node groupis increasing, etc.

Finally, shortly before the end of the aggregation time period 260 b, anadditional change C6 occurs at approximately time T20, prompted by oneof the computing nodes of the group failing or otherwise becomingunavailable. A corresponding event E9 occurs at approximately the sametime as the change C6, in which the PES terminates the unavailablecomputing node as needed. In this example, rather than immediatelyinitiating the providing of a replacement computing node, however, thePES determines to wait until the impending second set H2 ofharmonization activities are initiated at time T21 (e.g., based on thesmall amount of time remaining until time T21, based on such deferral ofproviding replacement computing nodes being the default action for anycomputing nodes that become unavailable while in use, etc.), since it ispossible that the quantity of computing nodes in the computing nodegroup will be reduced based on those harmonization activities and thereplacement computing node will not be needed. In other embodiments, thePES may instead immediately initiate the providing of the replacementcomputing node (e.g., based on not deferring the replacement ofunavailable computing nodes in any circumstances; based on not deferringthe replacement of unavailable computing nodes in this circumstance dueto the desired computing node quantity being likely to increase ratherthan decrease as part of the harmonization activities H2; based on notdeferring the replacement of unavailable computing nodes in thiscircumstance due to other factors, such as the user having already paidfor eight computing nodes until later time T3; etc.). In this example,in a manner similar to that previously discussed with respect to eventsE2 and E3, the event E9 is separated into two distinct events E9 a andE9 b (not shown separately) that occur at or near the same time. Thus,the PES records an event E9 a at time T20 in this example thatcorresponds to the computing node being detected to have becomeunavailable, and the PES automatically initiates activities at time T20to terminate the unavailable computing node, such as to directly causethe change C6. Furthermore, in this example, the termination of theunavailable computing node that occurs as the result of event E9 a isitself treated as a separate event E9 b at approximately time T20 thatinitiates a request for an additional computing node for the group touse as a replacement computing node, such as to maintain the desiredcomputing node quantity 215 at the desired quantity 215 a of eightcomputing nodes. In this example, the request of event E9 b is handledin a manner similar to events E4-E6, in being deferred until theharmonization activities H2 that will be performed at the end of thecurrent aggregation time period 260 b, although in other embodiments andsituations such providing of a replacement computing node may instead beinitiated immediately. In addition, in this example, the officialrecorded computing node quantity 225 is not updated to correspond tochange C6, in a manner similar to that previously described with respectto changes C4 and C5, although in other embodiments the officialrecorded computing node quantity 225 may be updated to reflect suchchanges.

Accordingly, at the end of the second aggregation time period 260 b, theharmonization activities H2 are initiated, and in this case do result indynamic modifications to the computing node group. In particular, thePES in this example aggregates the requested modifications correspondingto the various requested desired computing node quantities 215 b, 215 c,215 d and 215 a for events E4, E5, E6 and E9 b, and determines to make adynamic aggregated quantity modification to the prior desired computingnode quantity 215 a of eight, with the dynamic aggregated quantitymodification in this example being an increase of four additionalcomputing nodes (e.g., based on taking the largest of the requestedquantity modifications), so as to correspond to an updated currentdesired computing node quantity 215 e of twelve. In other embodiments,the dynamic aggregated quantity modification may be determined in othermanners, such as to select the dynamic aggregated quantity modificationso as to maintain the prior desired computing node quantity 215 a inaccordance with event E9 b (e.g., to result in an increase of onecomputing node from the current actual computing node quantity), to takea smallest of the requested quantity modifications of events E4-E6(e.g., to result in an increase of two computing nodes from the currentdesired computing node quantity), to take an average of the requestedquantity modifications of events E4-E6 (e.g., to result in an increaseof three computing nodes from the current desired computing nodequantity), to take an accumulation of the requested quantitymodifications of events E4-E6 (e.g., to result in an increase of ninecomputing nodes from the current desired computing node quantity), totake a highest priority of the requested quantity modifications ofevents E4-E6 (e.g., to result in an increase of three computing nodesfrom the current desired computing node quantity if, for example, theuser instruction of E4 is considered to be higher priority than thetrigger satisfactions of events E5 and E6), to take a first requested orlast requested of the requested quantity modifications of events E4-E6(e.g., to result in an increase of four or three computing nodes fromthe current desired computing node quantity, respectively), etc. In someembodiments and situations in which a user instruction event has higherpriority than trigger satisfaction events (e.g., if the user instructionalways overrides any requested dynamic modifications from triggersatisfaction events), the PES may further prevent additional triggersatisfaction events from occurring during an aggregation time periodafter a user instruction event is received, such as to in this exampleignore (or never determine) any trigger satisfaction events that occurafter the user instruction is received for event E4. Furthermore, inthis example, the current actual and official recorded computing nodequantities 220 and 225 differ, at seven and eight computing nodes,respectively, at the time of the harmonization activities H2. Therefore,as part of the harmonization activities H2, the PES initiates theproviding of five additional computing nodes for the computing nodegroup, to raise the current actual computing node quantity of seven tothe new updated desired computing node quantity of twelve, and furtherupdates the official recorded computing node quantity 225 to match thenew updated desired computing node quantity of twelve. Thus, in thisexample, the capacity availability change of 5 computing nodes isindirectly caused by replacing the one computing node that becameunavailable with respect to change C7, in accordance with event E9 b, aswell as by the increase of 4 additional computing nodes corresponding tothe determined aggregated quantity modification from the events E4, E5and E6.

During the third aggregation time period 260 c, additional events andcomputing node capacity availability changes further occur. Inparticular, a change C7 is completed at approximately time T22, in whichthe currently available computing node quantity is increased by five toa total of twelve computing nodes, to reflect the five additionalcomputing nodes whose availability was initiated at time T21, and withthe actual computing node quantity 220 being updated accordingly. As isshown in timeline graph 205, the aggregate average CPU utilization 205 aand the aggregate average network bandwidth utilization 205 b bothdecrease after the change C7, with the aggregate average CPU utilization205 a dropping quickly. In particular, in this example, the aggregateaverage CPU utilization 205 a eventually drops below a threshold of 20%corresponding to a criterion specified for trigger 250 b, causing anevent E8 to occur that includes a determination that trigger TR-3 250 bhas been satisfied, as is graphically illustrated in the timeline graph205 with a second black oval on the aggregate average CPU utilization205 a line. The trigger satisfaction initiates a request to decrease thedesired computing node quantity 215 by 2 computing nodes, for a totalrequested desired computing node quantity 215 f of ten (or alternativelycould request that the updated desired computing node quantity be set toten, regardless of the current desired computing node quantity).

Finally, an additional change C8 occurs before the end of theaggregation time period 260 c at approximately time T28, prompted by oneof the computing nodes of the group failing or otherwise becomingunavailable. A corresponding event E10 occurs at approximately the sametime as the change C8, in which the PES terminates the unavailablecomputing node as needed. In a manner similar to that for change C6, thePES opts in this example to wait until the third set H3 of harmonizationactivities are initiated at time T31 rather than immediately initiate areplacement for the unavailable computing node (e.g., since event E8makes it possible or likely that the desired quantity of computing nodesin the computing node group will be reduced then), although in otherembodiments the PES may instead immediately initiate the providing ofthe replacement computing node. In particular, in a manner similar tothat previously discussed with respect to events E2, E3 and E9, theevent E10 is separated into two distinct events E10 a and E10 b (notshown separately) in this example that occur at or near the same time.Thus, the PES records an event E10 a at time T28 in this example thatcorresponds to the computing node being detected to have becomeunavailable, and the PES automatically initiates activities at time T28to terminate the unavailable computing node, such as to directly causethe change C8. Furthermore, in this example, the termination of theunavailable computing node that occurs as the result of event E10 a isitself treated as a separate event E10 b at approximately time T28 thatinitiates a request for an additional computing node for the group touse as a replacement computing node, such as to maintain the desiredcomputing node quantity 215 at the desired quantity 215 e of twelvecomputing nodes. In this example, the request of event E10 b is handledin a manner similar to event E9 b, in being deferred until theharmonization activities H3 that will be performed at the end of thecurrent aggregation time period 260 c, although in other embodiments andsituations such providing of a replacement computing node may instead beinitiated immediately. In addition, in this example, the officialrecorded computing node quantity 225 is not updated to correspond tochange C8, in a manner similar to that previously described with respectto changes C4-C6, although in other embodiments the official recordedcomputing node quantity 225 may be updated to reflect such changes.

Accordingly, at the end of the third aggregation time period 260 c, theharmonization activities H3 are initiated, and do result in changes tothe computing node group. In particular, the PES in this exampleaggregates the requested modifications corresponding to events that haveoccurred during the time period 260 b, which in this example is onlyevent E8 and E10 b, and determines to make a dynamic aggregated quantitymodification to the prior desired computing node quantity 215 e oftwelve, which in this example is a decrease of two computing nodes, soas to correspond to an updated current desired computing node quantity215 g of ten. Furthermore, the current actual and official recordedcomputing node quantities 220 and 225 differ, at eleven and twelvecomputing nodes, respectively, at the time of the harmonizationactivities H3. Therefore, as part of the harmonization activities H3,the PES initiates a change in the current official computing nodequantity of twelve to reach the current desired actual computing nodequantity of ten, in light of the current actual computing node capacityof eleven, so as to remove one of the existing computing nodes from thecomputing node group (e.g., to terminate the execution of the program onthe removed computing node, optionally after completing some or allactions that it has already begun, and to make that computing nodeavailable for other future use by other users). Thus, in this example,the capacity availability change of removing one of the existingcomputing nodes at time T31 is indirectly caused by the terminating ofthe one group computing node with respect to event E10 b, as well as bythe requested decrease of 2 computing nodes corresponding to the eventE8. The PES further updates the official recorded computing nodequantity 225 to match the new updated desired computing node quantity often. Shortly after time T31, a final change C9 is completed atapproximately time T32, in which the currently available computing nodequantity is decreased by one to a total of ten computing nodes, toreflect the computing node whose removal from the computing node groupwas initiated at time T31, and with the actual computing node quantity220 being updated accordingly. In other embodiments, the change C9 mayoccur substantially immediately at time T31 upon the determination tomake the change, such as if the computing node to be removed isimmediately withdrawn from further activities for the computing nodegroup, even while the removed computing node is temporarily executingand available to continue performing operations.

In addition, while the harmonization activities H1, H2 and H3 areillustrated in this example as occurring at a single point in time, itwill be appreciated that some or all harmonization activities mayactually take place over a period of time, and further may have othereffects that last over a period of time. For example, in at least someembodiments, some or all changes to program execution capacity of acomputing node group that are initiated by the PES may result in atemporary lockdown period in which at least some other types of eventsor changes may not be allowed. The PES-indicated program executioncapacity changes that may cause such a lockdown may include, forexample, program execution capacity increases and/or decreases that areinitiated as part of the harmonization activities (e.g., to add newcomputing nodes, to remove existing computing nodes, etc.), and/orprogram execution capacity changes that are initiated immediately inresponse to a PES determination (e.g., a computing node failure, receiptof a user instruction, satisfaction of a user-specified trigger, etc.).Such lockdowns may have durations of various types (e.g., until aspecified result or occurrence, such as until a computing node beingadded has become available; a specified length of time, such as theaverage or expected time for a computing node being added to becomeavailable; etc.), and in some embodiments may vary based on the type ofPES-initiated change. During such lockdowns, at least some types ofevents or changes may not be allowed, such as to not allowuser-specified triggers to be satisfied (or by ignoring any such triggersatisfactions) during the lockdown duration, and/or to not allow userinstructions to be accepted (or by ignoring any such user instructions)during the lockdown duration. Furthermore, in some embodiments, a userassociated with a computing node group may similarly specify a cooldownperiod that operates in a similar manner to that of a lockdown, such asa cooldown period of a specified amount of time that follows a lockdown,or instead takes effect at other times. As with the lockdown, during thecooldown period, at least some types of events or changes may not beallowed, such as to not allow user-specified triggers to be satisfied(or by ignoring any such trigger satisfactions) during the cooldownperiod. It will be appreciated that users and/or the PES may controlmodifications to computing node groups in other manners in otherembodiments.

Thus, in this manner, the PES operates to manage the program executioncapacity provided to a user by a group of computing nodes, including tomake various dynamic modifications to the computing node group based ona variety of circumstances. It will be appreciated that the events andchanges illustrated in FIG. 2A are provided for illustrative purposesonly, and that actual events, changes and other operations of the PESmay differ in other embodiments and situations.

FIG. 2B illustrates an example of techniques for automaticallyattributing causation information to the dynamic computing node quantitymodifications previously discussed with respect to FIG. 2A, such astechniques that may be automatically performed by an embodiment of aPESSM module. In particular, FIG. 2B illustrates two example databasetable data structures that store information corresponding to some ofthe information illustrated in FIG. 2A, with the example table 280 ofFIG. 2B storing various information related to the example computingnode capacity availability changes that occurred to the examplecomputing node group of FIG. 2A (referred to as “Group1” in FIG. 2B),and with example table 290 of FIG. 2B storing various informationrelated to the example events that occurred for the example computingnode group Group1 of FIG. 2A.

Example table 280 includes a variety of rows or entries 285 a-285 k thateach corresponds to one of the example changes C1-C10 discussed withrespect to FIG. 2A, with various fields or columns 280 a-280 xillustrated for each row. In particular, in this example, each rowincludes a unique identifier (“ID”) 280 a, an ID 280 b of the applicablecomputing node group to which the change corresponds, a type 280 c ofthe program execution capacity change, an indication 280 d of theintended result of the program execution capacity change, variousdetails 280 e about the change, start and end times 280 f and 280 g ofthe change, an aggregation time period 280 h during which the changetakes place or otherwise with which the change is associated (which inthis example is referred to based on the harmonization activitiesperformed at the end of the aggregation time period), a change-eventnexus ID 280 x, and optionally various other information. As oneillustrative example, row 285 a corresponds to change C1 for Group1,which corresponds to a change in the computing node quantity of Group1involving an increase of 4 computing nodes that is completed at time T02during aggregation time period H1. The other rows include similarinformation.

Example table 290 includes a variety of rows or entries 295 a-295 m thateach corresponds to one of the example events E1-E10 discussed withrespect to FIG. 2A, with various fields or columns 290 a-290 xillustrated for each row. In particular, in this example, each rowincludes a unique identifier (“ID”) 290 a, an ID 290 b of the applicablecomputing node group to which the event corresponds, a type 290 c of theevent, information 290 d about the source of the event, a time 290 e ofthe event, an aggregation time period 290 f during which the event takesplace or otherwise with which the event is associated (which in thisexample is referred to based on the harmonization activities performedat the end of the aggregation time period), a change-event nexus ID 290x, and optionally various other information (e.g., a particular userassociated with received user instructions). As one illustrativeexample, row 295 h corresponds to event E4 for Group1, which correspondsto a received user instruction at time T19 during aggregation timeperiod H2 that requests that the desired computing node quantity ofGroup1 be increased by 3 computing nodes. The other rows include similarinformation.

In the example of FIG. 2B, the change-event nexus ID information 280 xand 290 x of tables 280 and 290 reflects causation information that isattributed to reflect relationships between changes and events. Inparticular, in this example, if rows in the tables 280 and 290 share acommon value for the change-event nexus ID information 280 x and 290 x,it reflects attributed causation between the corresponding events andchanges. As one example, as previously discussed with respect to FIG.2A, event E3 b corresponds to the providing of a replacement computingnode for a computing node that became unavailable, which is directlyattributable to change C5 corresponding to the resulting addition of acomputing node to the group. Accordingly, row 295 e (corresponding toevent E3 b) and row 285 g (corresponding to change C5) share a commonnexus identifier.

Conversely, change C7, which corresponds to adding five computing nodesthat is initiated during the harmonization activities H2, does not havea single event that is directly attributable to that change. Inparticular, events E4, E5 and E6 each requested dynamic modifications toGroup1, which were aggregated and used in combination to prompt theadding of four of the five computing nodes for change C7. As such, rows295 f-295 h (corresponding on events E4-E6) and row 285 i (correspondingto change C7) all share a common nexus identifier. Furthermore, in theexample of FIG. 2A, one of the five computing nodes that was added aspart of change C7 was a replacement node for the computing node thatbecame unavailable with respect to change C6, with the initiation of thereplacement computing node corresponding to event E9 b. Accordingly, row295 j (corresponding to event E9 b) and row 285 i (corresponding tochange C7) also share the same common nexus identifier. In otherembodiments, if the providing of the replacement computing node wasinitiated in event E9 b separately from the providing of the fouradditional computing nodes corresponding to events E4-E6, such as toimmediately replace the unavailable computing node, the correspondingchange for that would occur before the end of time period 260 b and theresulting harmonization activities H2, and would be shown separately intable 280 with event E9 b and that additional change sharing a distinctcorresponding nexus identifier. In addition, while the five computingnodes that are added for change C7 are shown in an aggregated manner ina single row in table 280, in order embodiments each computing nodebeing added may be represented in a separate row 285 of the table 280,and if so would each share the same nexus identifier N7 that iscurrently illustrated for only row 285 i of aggregated change C7.

Furthermore, in this example, changes are tracked in table 280 thatcorrespond not only to dynamic modifications to computing node quantitybased on dynamic user instructions and/or satisfied user triggers, butalso based on inadvertent changes to computing node quantity (e.g., dueto computing nodes failing or otherwise becoming unavailable). Suchchanges are detected based on monitoring activities, and are illustratedas events that trigger additional changes (e.g., the providing ofreplacement computing nodes), although in other embodiments may behandled in other manner, such as to not track such changes and/or to nottreat system-initiated replacement activities as events. Furthermore,while the change-event nexus information in the tables 280 and 290 doesnot distinguish between changes that are directly attributable to one ormore corresponding events (e.g., events that cause those changes to beinitiated immediately) and changes that are indirectly attributable toone or more corresponding events (e.g., events that are aggregatedtogether and in combination cause those changes to be initiated, such asduring harmonization activities), in other embodiments such informationmay further be tracked.

As previously noted, the attribution of responsibility for particulardynamic program execution capacity modifications to a computing nodegroup may provide various benefits, including in providing explanationsto a user associated with the computing node group of why changesoccurred. Such responsibility attribution information may further begenerated and/or used in response to various types of queries receivedfrom users or other sources, such as a request to identify whichevent(s) are the cause of a particular indicated program executioncapacity modification or other change in availability of one or morecomputing nodes of a group, and/or of which program execution capacitymodification(s) or other computing node group availability change(s) arecaused by one or more indicated events. As discussed with respect toFIG. 2B, the nexus information illustrated in tables 280 and 290provides one mechanism for tracking and providing responsibilityattribution information. For example, with respect to the addition of 5computing nodes corresponding to change C7 in FIG. 2A, the user may wantto know why 5 computing nodes were added (e.g., particularly in light ofthe user instruction of event E4 to add 3 computing nodes). By using thenexus information illustrated in FIG. 2B, the PES or other system mayeasily automatically generate a human-readable explanation. For example,in response to a user request as to the cause of adding the fourcomputing nodes corresponding to change C1 of FIG. 2A, the PES or othersystem may, for example, indicate the following: “Change C01 wasdirectly caused by event E01 at time T1. Event E01 was a request fromUser UUU to initiate a group of 8 computing nodes.” This information maybe generated based on using the nexus information 280 x from row 285 acorresponding to change C1 to identify row 295 a of the table 290corresponding to event E1, and extracting and formatting informationfrom rows 285 a and 295 a in a desired manner (e.g., in a format basedon the user request or prior user preferences, in a format based on PESdefaults, etc.). Similarly, in response to a user request as to theeffect of the user instruction E1 to initiate the group of eightcomputing nodes, the PES or other system may, for example, indicate thefollowing: “Event E01 at time T1 directly caused Changes C01, C02 andC03. Change C01 was the addition of 4 computing nodes that was initiatedat time T1 and ended at time T2. Change C02 was the addition of 3computing nodes that was initiated at time T1 and ended at time T3.Change C03 was the addition of 1 computing node that was initiated attime T1 and did not complete.” As another example, in response to a userrequest as to the cause of adding the five computing nodes correspondingto change C7 of FIG. 2A, the PES or other system may, for example,indicate the following: “Change C07 was indirectly caused by events E05,E06, E04 and E09 b that occurred during the time period from T11 to T21.Event E05 was a request for 4 additional computing nodes based onsatisfaction of trigger TR-1. Event E06 was a request for 2 additionalcomputing nodes based on satisfaction of trigger TR-N. Event E04 was arequest for 3 additional computing nodes based on a dynamicuser-supplied instruction from User UUU. Event E09 b was a request for 1replacement computing node based on the automated termination of acomputing node of the group that became unavailable.” In addition to orinstead of such text strings, responsibility attribution information mayfurther be generated in various other forms, such as automated reports(e.g., in tables, charts, etc.), and in a periodic or other scheduledmanner. It will be appreciated that such responsibility attributioninformation may be generated and used in a variety of other manners inother embodiments.

It will be appreciated that the information regarding events and changesillustrated in FIG. 2B is provided for illustrative purposes only, andthat the information that is stored and the storage of such informationmay be performed in a variety of other manners in other embodiments.Furthermore, the PES may store a variety of additional types ofinformation about users and computing node groups in other embodiments,such as to have additional tables that store information aboutuser-defined triggers, about monitored performance measurementinformation, about user accounts, etc.

In addition, the preceding examples of FIGS. 2A and 2B are provided forillustrative purposes, and other embodiments may differ in various waysfrom the examples. For example, although the program execution capacityis measured and modified based on a quantity of computing nodes in theseexamples, such as if the various available computing nodes are treatedas being equivalent (e.g., having equivalent computing resources), otherembodiments may be provided where various of the available computingnodes may be of different types with varying characteristics (e.g.,different amounts of processing capacity, memory, platformspecification, etc.), and/or in which program execution capacity istracked in manners other than computing node quantity. In some suchembodiments, various of the requests may include indications of one ormore specific types of the computing nodes for use in groups ofcomputing nodes selected to execute programs associated with therequests, and those requests may only be fulfilled on the correspondingspecified type of computing node.

In addition, in at least some embodiments, a PES or other system mayfurther perform other types of activities as part of managing groups ofcomputing nodes. For example, as previously noted, the PES or othersystem may determine to add or remove computing nodes from a computingnode group at various times, including the following: duringharmonization activities at the end of an aggregation period of time inresponse to one or more triggers satisfied during the time period and/oruser instructions received during the time period; at any time inresponse to a user instruction or trigger satisfaction that is specifiedor determined to have immediate effect; at any time in response toautomated activities of the PES or other system, such as to replace afailed computing node and/or to terminate and remove a computing nodewhose ongoing operation is inhibited; etc. In at least some embodiments,the PES or other system may further consider other factors whendetermining the exact timing for at least some such computing node groupmodifications. For example, in situations in which a user will be heldresponsible for the use of a given quantity of computing nodes for agiven period of time (e.g., has already been charged for X computingnodes for the next hour), such information may be used to determinetiming related to at least some types of modifications to the computingnode group. If the PES or other system determines to reduce the quantityof computing nodes in the group below X, the PES or other system maydetermine to wait until near or at the end of the given time period(e.g., the end of the hour for which the user has already been charged)before actually reducing the computing node quantity below X. Similarly,if a computing node of a group fails near the time when newharmonization activities will be performed (e.g., near the end of aperiod of time during which requested computing node quantitymodifications are being aggregated), the PES or other system maydetermine to update the actual node quantity and optionally the officialrecorded node quantity to reflect the failure, but to not immediatelyinitiate the addition of a replacement computing node—in this manner, ifthe aggregation of the requested computing node quantity modificationsduring the harmonization activities determine to reduce the computingnode quantity for the group by one or more computing nodes, the failedcomputing node may be used as one such reduction rather than terminatingan executing computing node of the group.

In some embodiments, the PES or other system may manage multipledistinct types of modifications for a computing node groupsimultaneously. As one example, a computing node group may be beingmanaged at a current desired computing node quantity using computingnodes at a single location, but the user may decide to specify a secondtype of metric for the computing node group, such that a specifieddesired subset of the computing nodes of the group be located at adistinct second location. If so, the PES or other system may operate tomeet the desired value for the second metric in various manners,including by incrementally adding any new computing nodes to the groupat the second location and incrementally removing any existing computingnodes from the group at the first location until the desired value forthe second metric is achieved, or alternatively immediately terminatingexisting computing nodes from the group at the first location to enablereplacement computing nodes to be added to the group at the secondlocation. Such incremental additions and/or removals may be triggered inany of the manners discussed in greater detail elsewhere, including toreplace unavailable computing nodes, modify computing node quantity inresponse to dynamically specified user instructions and/or satisfieduser-specified triggers, etc. In addition, while computing node quantityand location are simultaneously being balanced in this example, avariety of other types of changes may be performed in a similar manner(e.g., to change existing computing nodes from a first type to a secondtype, such as based on the different types having differing associatedamounts of computing resources). Furthermore, while program executioncapacity modifications are made in some examples by changing computingnode quantity, in other embodiments such capacity modifications may bemade by changing the program execution capacity of one or more computingnodes of the group (e.g., by replacing a first computing node with asecond computing node that has more or less of one or more types ofcomputing resources of interest, by modifying the amount of one or morecomputing resources that are available to one of the computing nodesthat are already part of the group, etc.).

As previously discussed, various types of functionality may be providedand used by a PES in various embodiments, and the functionality may beprovided in various ways. For example, in some embodiments, programexecution capacity available from a PES may include multiple computingnodes for executing programs on behalf of users, such as via multiplephysical computing machines interconnected via one or more networks orother data exchange mediums that are capable of transmitting databetween the computing machines. At least some of the computing machinesmay in some embodiments each include sufficient computing-relatedresources to execute multiple programs simultaneously (e.g., sufficientwriteable memory, non-volatile storage, CPU cycles or other CPU usagemeasure, network bandwidth, swap space, etc.), and at least some of thecomputing machines in some such embodiments may each host multiplevirtual machine computing nodes that each may execute one or moreprograms on behalf of a distinct user. Furthermore, in variousembodiments, a PES may execute various types of programs on behalf ofmultiple users. For example, such programs executed on behalf of usersmay include one or more operating systems, applications (e.g., serversand/or other software applications), utilities, libraries, etc. Inaddition, in at least some embodiments, such programs may includeexecutable software images, such as virtual machine images that arebootable or otherwise loadable on one or more virtual machine computingnodes, and that each may include operating system software, software forone or more application programs, and/or configuration information, etc.

In at least some embodiments, the execution of one or more programs on agroup of one or more computing nodes by a PES may be initiated inresponse to a current execution request for immediate execution of thoseprograms. Alternatively, the initiation may be based on a previouslyreceived program execution request that scheduled or otherwise reservedthe then-future execution of those programs for the now-current time.Program execution requests may be received in various ways, such asdirectly from a user (e.g., via an interactive console or other GUIprovided by the program execution service), or from an executing programof a user that automatically initiates the execution of one or moreother programs or other instances of itself (e.g., via an API providedby the program execution service, such as an API that uses Webservices). Program execution requests may include various information tobe used in the initiation of the execution of one or more programs, suchas an executable or other copy of a program to be executed, anindication of a program that was previously registered or otherwisesupplied for execution, and a number of instances of the program thatare to be executed simultaneously (e.g., expressed as a single desirednumber of instances, as a minimum and maximum number of desiredinstances, etc.), as well as a variety of other types of preferencesand/or requirements for execution of one or more programs (e.g.,resource allocation, geographical and/or logical location for execution,proximity of execution to other programs and/or computing nodes,timing-related criteria, etc.).

After receiving a request to execute one or more instances of a programat an indicated time, the PES may determine one or more computing nodesto use in a group for executing the program instances. In someembodiments, the determination of the computing nodes to be used isperformed at the time of the request even if for future execution. Inother embodiments, the determination of the computing nodes to be usedfor future execution of one or more program instances may be deferred toa later time, such as at the future time of execution based oninformation that is then available. In some embodiments, thedetermination of which computing nodes to use for execution of one ormore programs on behalf of a user may be made prior to a request toexecute, such as at a time when a user subscribes and/or registers touse the PES, and/or at another time prior to a request to executeprograms for a user. For example, in some such embodiments, one or morecomputing nodes may be associated with a user for a period of time, suchthat programs may be executed on behalf of that user on the associatedcomputing nodes at any time during that period, such as at any time arequest is received to execute software for the user during the period.In addition, in some embodiments, the determination of which computingnodes to use to execute programs on behalf of a user may be made whenone or more computing nodes and/or computing resources of one or morecomputing nodes become available for executing programs for the user,such as, for example to execute programs of one or more pending requestson one or more computing nodes at a time when the computing nodes areunused and/or are otherwise available for executing the programs.

The determination of which computing nodes to use for execution of eachprogram copy or instance may be made in a variety of ways, includingbased on any preferences and/or requirements specified in the request orotherwise specified for the program and/or associated user (e.g., at atime of registration, etc.). For example, if criteria are determined forpreferred and/or required resources for execution of a program instance(e.g., memory and/or storage; CPU type, cycles or other performancemetric; network capacity; platform type, etc.), the determination of anappropriate computing node to execute a program instance may be based atleast in part on whether a computing node has sufficient resourcesavailable to satisfy those resource criteria. Additional details relatedto executing and configuring programs to execute on a program executionservice are included in pending U.S. patent application Ser. No.11/395,463, filed Mar. 31, 2006 and entitled “Managing Execution ofPrograms by Multiple Computing Systems,” which is hereby incorporated byreference in its entirety.

In some embodiments, fees may be associated with the use of a PES, suchthat the PES may execute programs on behalf of a user in exchange forpayment of one or more fees by that user. For example, in someembodiments, fees may be charged to a user based on an amount and/ortype of program execution capacity allocated for executing one or moreprograms on behalf of a user, such as based on one or more of a numberof processing units, an amount of memory, an amount of storage, anamount of network resources, etc., allocated for executing programs ofthe user. In some embodiments, fees may be based on other factors, suchas various characteristics of the computing resources used to executeprograms, such as, for example, based on CPU capabilities orperformance, platform type (e.g., 32-bit, 64-bit, etc.), etc. In someembodiments, fees may be charged on the basis of a variety of usefactors, such as a price per use of the service, a price per unit oftime that computing services are used, a price per storage used, a priceper data transferred in and/or out, etc. In at least some embodiments,as discussed in more detail below, fees may be based on various otherfactors, such as various properties related to executing programs (e.g.,continuity of execution, fault tolerance, etc.). In at least someembodiments, a provider of a PES may offer one or more of various tiers,types and/or levels of services or functionality for executing programson behalf of multiple users, and in some such embodiments, various feesmay be associated with the various tiers, types and/or levels ofservices. In addition, for example, tiers may be used for a specifictype of functionality provided by a PES, such as to charge fees at afirst tier for a first quantity of program execution capacityfunctionality (e.g., up to a specified first threshold of computingnodes being used), to charge fees at a second tier (e.g., a lower pricetier) for a second quantity of program execution capacity functionality(e.g., above the specified first threshold and up to a specified secondthreshold of computing nodes being used), etc. Additional detailsrelated to various fees associated with a program execution service areincluded in pending U.S. patent application Ser. No. 11/963,331, filedDec. 21, 2007 and entitled “Providing Configurable Pricing for Executionof Software Images,” which is hereby incorporated by reference in itsentirety.

Furthermore, various other types of functionality may be provided andused by a PES in various embodiments, as discussed in greater detailelsewhere.

FIG. 3 is a block diagram illustrating an example embodiment of a systemsuitable for performing techniques to manage groups of computing nodesfor multiple users. In particular, FIG. 3 illustrates a server computingsystem 300 suitable for providing at least some functionality of aprogram execution service, as well as various client computing systems350 that may be used by users of the program execution service,computing nodes 360 that may be used by the program execution service,and other computing systems 380. In the illustrated embodiment, theserver computing system 300 has components that include a CPU 305,various I/O components 310, storage 320, and memory 330. The illustratedI/O components include a display 311, a network connection 312, acomputer-readable media drive 313, and other I/O devices 315 (e.g., akeyboard, a mouse, speakers, etc.). In addition, the illustrated usercomputing systems 350 have components similar to those of servercomputing system 300, including a CPU 351, I/O components 352, storage354, and memory 357. The other computing systems 380 and computing nodes360 may also each include components that are similar to some or all ofthe components illustrated with respect to server computing system 300,but such components are not illustrated in this example for the sake ofbrevity.

An embodiment of a Program Execution Service System Manager module 340is executing in memory 330, and it interacts with computing systems 350and 380 and computing nodes 360 over the network 390 (e.g., via theInternet and/or the World Wide Web, via a private cellular network,etc.). In this example embodiment, the PESSM 340 includes functionalityrelated to managing use of multiple computing nodes 360 by various users(not shown) interacting with user computing systems 350, such as inconjunction with a program execution service managed by the PESSM 340.The other computing systems 350 and 380 and computing nodes 360 may beexecuting various software as part of interactions with the PESSM. Forexample, user computing systems 350 may be executing software in memory357 to interact with PESSM 340 (e.g., as part of a Web browser orspecialized client-side application program), such as to configureand/or request execution of programs on behalf of the users of thosesystems on one or more computing nodes 360 in various ways, as well asto perform various other types of actions, as discussed in greaterdetail elsewhere. Various information related to the functionality ofthe PESSM module 340 may be stored in storage 320, such as information322 related to configuration, execution and/or registration forexecuting programs on behalf of multiple users, information 324 relatedto program execution capacity modifications for computing node groups(e.g., information about predefined user-specified triggers; informationabout dynamically specified capacity modification instructions fromusers; information about computing node performance measurements andother information related to determining if specified triggers aresatisfied; current values for desired, actual and official recordedcomputing node quantities for groups; etc.), and information 326 relatedto attribution of causation for particular program execution capacitymodifications for particular computing node groups (e.g., by listing atleast some events that occur and at least some changes to computingnodes of groups, and associating particular events with particularchanges, such as in a manner similar to that discussed in FIG. 2B andelsewhere).

After the PESSM module 340 receives requests (or other indications) toexecute one or more programs on a group of one or more computing nodes360, the PESSM module 340 selects the one or more computing nodes forthe group, and initiates execution of those programs on those computingnodes 360. In addition, the PESSM module 340 may further interact withcomputing nodes 360 to later terminate execution of initiated programson the computing nodes, to migrate one or more of the programs to one ormore other computing nodes 360 or computing systems 380, etc. Thecomputing nodes 360 may have various forms in various embodiments, suchas to include a number of physical computing systems and/or a number ofvirtual machines executing on one or more physical computing systems. Insome embodiments, the server computing system 300 and computing nodes360 may be part of a data center or other group of co-located computingsystems, or may otherwise be computing nodes of a private network. Inaddition, in some embodiments, the PESSM module 340 may interact withone or more other computing systems 380 to initiate or terminateexecution of one or more programs on those computing systems, such as ifthe computing systems 380 are provided by one or more third-partyparticipants who are providing computing nodes for use by other users.In some embodiments, the PESSM module 340 may further or instead manageaccess to one or more types of computing-related resources or servicesother than program execution services (e.g., persistent or temporarydata storage services, messaging services, database services, etc.).

In addition, an embodiment of a Program Execution Service CapacityMaintenance Manager module 345 is executing in memory 330, and itinteracts in this embodiment with computing nodes 360 over the network390. In particular, in this example embodiment, the PESCMM module 345includes functionality related to monitoring or otherwise interactingwith one or more of the computing nodes 360 to track use of thosecomputing nodes, such as to determine current actual program executioncapacity of a computing node group and/or to determine currentperformance characteristics corresponding to some or all computing nodesof a computing node group. As previously noted, such information may bestored on storage 320 and/or elsewhere, and may be used by the modules340 and 345 in various manners. For example, in some embodiments, if themodule 345 discovers that a computing node has failed or otherwisebecome unavailable (e.g., as part of provisioning or otherwiseinitializing the computing node to be used as part of a computing nodegroup, after the computing node has been in use as part of a computingnode group, etc.), the module 345 may automatically take actions toreplace the unavailable computing node with a new computing node. Inother embodiments, the module 345 may instead not perform some or all ofthe monitoring of the computing nodes, such as if the module 345 insteadobtains information from another source about current actual programexecution capacity of a computing node group and/or current performancecharacteristics corresponding to some or all computing nodes of acomputing node group, and then uses that information to maintain programexecution capacity for the computing node group as appropriate.

It will be appreciated that computing systems 300, 350 and 380 andcomputing nodes 360 are merely illustrative and are not intended tolimit the scope of the present invention. The computing systems and/ornodes may instead each include multiple interacting computing systems ordevices, and the computing systems/nodes may be connected to otherdevices that are not illustrated, including through one or more networkssuch as the Internet, via the Web, or via private networks (e.g., mobilecommunication networks, etc.). More generally, a computing node or othercomputing system may comprise any combination of hardware or softwarethat may interact and perform the described types of functionality,including without limitation desktop or other computers, databaseservers, network storage devices and other network devices, PDAs,cellphones, wireless phones, pagers, electronic organizers, Internetappliances, television-based systems (e.g., using set-top boxes and/orpersonal/digital video recorders), and various other consumer productsthat include appropriate communication capabilities. In addition, thefunctionality provided by the illustrated modules 340 and/or 345 may insome embodiments be distributed in additional modules. Similarly, insome embodiments some of the functionality of the modules 340 and/or 345may not be provided and/or other additional functionality may beavailable.

It will also be appreciated that, while various items are illustrated asbeing stored in memory or on storage while being used, these items orportions of them may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other manners, such as atleast partially in firmware and/or hardware, including, but not limitedto, one or more application-specific integrated circuits (ASICs),standard integrated circuits, controllers (e.g., by executingappropriate instructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the modules,systems and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network, or a portable media article to be readby an appropriate drive or via an appropriate connection. The systems,modules and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmissionmediums, including wireless-based and wired/cable-based mediums, and maytake a variety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

FIG. 4 is a flow diagram of an example embodiment of a Program ExecutionService System Manager routine 400. The routine may be provided by, forexample, execution of the PESSM modules 110 and 180 of FIGS. 1A and 1B,respectively, and/or the PESSM module 340 of FIG. 3, such as to assistin managing use of groups of computing nodes for users, as well as toperform other types of management operations in some situations. In thisillustrated embodiment, the routine 400 manages various aspects of useof a program execution service that provides program execution capacityfor executing programs on behalf of multiple users.

In the illustrated embodiment, the routine begins at block 405, whereinformation or a request is received. The routine continues to block 410to determine if the received request or information is related toinitiating execution of one or more programs on a group of computingnodes, such as a request from a user. If so, the routine continues toblock 415 to obtain information regarding the requested programexecution, such as an initial desired amount of program executioncapacity for the computing node group (e.g., a desired computing nodequantity), optionally one or more programs to be executed, andoptionally one or more user-specified capacity modification triggers. Asdiscussed elsewhere, in some embodiments, a user may select from one ormore of various types of computing nodes and/or may otherwise specifyvarious amounts and/or types of computing resources desired (e.g.,processing unit type/amount, memory amount, platform specification,etc.). In block 420, the routine then selects the computing nodes to beused for the group, and in block 425 initiates making those selectedcomputing nodes available for the user, such as by provisioning theselected computing nodes and optionally initiating execution of the oneor more programs to be executed. The routine also designates the desiredprogram execution capacity of the selected computing nodes of the groupas the initial official recorded program execution capacity for thegroup. When the computing nodes are available for use on behalf of theuser, the user may be notified of the availability in various manners,or in other embodiments the computing nodes may operate in an automatedmanner without further interaction by the user. The routine thencontinues to block 430 to store information related to the computingnode group, including any user-specified triggers for the group.

If it is instead determined at block 410 that a request to initiateexecution of a group of computing nodes is not received, the routineinstead continues to block 440 to determine whether a request isreceived related to modifying program execution capacity for an existingcomputing node group. If so, the routine continues to block 445 toreceive and store a dynamically specified user instruction related tomodifying program execution capacity for an indicated existing computingnode group. In the illustrated embodiment, the user instruction may beaggregated with other possible program execution capacity modificationrequests that occur during a current aggregation time period, andfurther processed during a next time that harmonization activities areperformed, such as with respect to blocks 465-477, although in otherembodiments at least some such user-specified modification requests mayinstead be performed immediately.

If it is instead determined at block 440 that a request to modifyprogram execution capacity for a group of computing nodes is notreceived, the routine instead continues to block 460 to determinewhether to currently perform periodic or otherwise recurrentharmonization activities with respect to the program execution capacityfor one or more computing nodes groups, such as at the end of a periodof time of aggregation of program execution capacity modificationrequests for such a computing node group. If so, the routine continuesto block 465 to determine one or more computing node groups for which tocurrently perform harmonization activities (e.g., computing node groupsfor which an aggregation period of time has ended and for which one ormore dynamic program execution capacity modifications have beenaggregated), and in block 467 selects the next determined computing nodegroup, beginning with the first. The routine then continues to block 470to execute a routine to perform recurrent capacity harmonizationactivities, with one example of such a routine being described ingreater detail with respect to FIG. 5. The routine next continues toblock 475 to execute a routine to perform activities related toattribution of causation for program execution capacity modificationsthat are performed in block 470, with one example of such a routinebeing described in greater detail with respect to FIG. 6. After block475, the routine continues to block 477 to determine whether there aremore determined computing node groups to process, and if so the routinereturns to block 467 to select the next such determined computing nodegroup.

If it is instead determined at block 460 not to currently performperiodic or otherwise recurrent harmonization activities with respect tothe program execution capacity for one or more computing nodes groups,the routine instead continues to block 480 to optionally perform one ormore other indicated operations. Such operations may include, forexample, one or more of the following: user requests related toperforming other types of program execution (if the provided programexecution service provides such other program execution types), such asto execute a single program on a single computing node; user-specifiedprogram execution capacity modification requests that are to beperformed immediately (e.g., a user instruction to terminate executionof a particular indicated computing node, such as if the computing nodeis not operating properly); user requests to specify additional triggersor otherwise to modify configuration information for an indicatedcomputing node group; user requests to immediately perform harmonizationactivities with respect to an indicated computing node group, such as inaddition to or instead of recurrent harmonization activities (e.g., ifharmonization activities are performed only upon user request); userrequests to obtain various status information related to one or morecomputing node groups with which the user is associated; requests toperform administrative-related activities for a user, such assubscription, registration, or payment operations; etc.

After blocks 430, 445, or 480, or if it is instead determined in block477 that additional determined computing node groups are not available,the routine continues to block 485 to optionally perform any userbilling (or reimbursement) activities based on the information orrequest received in block 405 or as is otherwise initiated (e.g.,periodically), such as to charge and/or collect fees from one or moreusers based on program execution functionality provided to the users.The routine may further optionally perform periodic housekeepingoperations as appropriate.

After block 485, the routine continues to block 495 to determine whetherto continue, such as until an explicit indication to terminate executionof the routine. If it is determined to continue, the routine returns toblock 405, and if not continues to block 499 and ends. It will beappreciated that additional types of activities may be performed in someembodiments and situations, such as to determine whether users areauthorized to perform particular requested operations, to immediatelyobtain payment from users for some types of requested operations, etc.In addition, while user requests and other operations are indicated inthe illustrated embodiment as being performed in a manner specific to aparticular computing node group and a particular associated user, inother embodiments some or all such operations may instead be appliedmore generally, such as to multiple computing nodes groups associatedwith a single user and/or from multiple users associated with one ormore computing node groups.

FIG. 5 is a flow diagram of an example embodiment of a RecurrentCapacity Harmonization routine 500. The routine may be provided by, forexample, execution of the PESSM modules 110 and 180 of FIGS. 1A and 1B,respectively, and/or the PESSM module 340 of FIG. 3, such as may beinitiated from block 470 of routine 400 in FIG. 4.

In the illustrated embodiment, the routine begins in block 505, where itobtains an indication of the currently selected computing node group forwhich capacity harmonization activities are to be performed. In block515, the routine then retrieves information about the official recordedprogram execution capacity for the selected computing node group, suchas based on the prior harmonization activities performed for theselected computing node group (or the initial official recorded programexecution capacity for the selected computing node group if this is thefirst harmonization activities that are performed), and/or based onsubsequent modifications to the official recorded program executioncapacity that may be performed by a PES Capacity Maintenance Managermodule (e.g., as described in greater detail with respect to FIG. 7).The routine then continues to blocks 520 and 525 to determine thecurrent actual and current desired program execution capacities of theselected computing node group, respectively. The determining of thecurrent actual program execution capacity may include, for example,retrieving information that was previously stored by a PES CapacityMaintenance Manager module as part of monitoring the selected computingnode group, although in other embodiments the routine 500 maydynamically determine the current actual capacity (e.g., by dynamicallyrequesting a PES Capacity Maintenance Manager module or other monitoringsource to provide that information).

The determining of the current desired program execution capacity inblock 525 may include, for example, retrieving information regarding anydynamically specified capacity modification instructions that have beenreceived from an associated user for the selected computing node groupduring the current aggregation period of time to which the harmonizationactivities correspond (e.g., as previously discussed with respect toblock 445 of FIG. 4), and regarding any user-specified triggers for theselected computing node group that have previously been determined tohave been satisfied during the current aggregation period of time.Alternatively, in some embodiments, the routine may instead retrievedetermined performance characteristics information for the selectedcomputing node group (e.g., information that was previously stored by aPES Capacity Maintenance Manager module as part of monitoring theselected computing node group, or information dynamically obtained byrequesting a PES Capacity Maintenance Manager module or other monitoringsource to provide that information) in order to currently determinewhether any user-specified triggers for the selected computing nodegroup are currently satisfied and/or were previously satisfied duringthe current aggregation period of time. After the various information isretrieved, the current desired program execution capacity may bedetermined by aggregating the one or more requested program executioncapacity modifications, as discussed in greater detail elsewhere, inorder to determine the resulting desired program execution capacityafter any such aggregated capacity modifications are made.

After block 525, the routine continues to block 530 to determine theactual changes to be made to the selected computing node group in orderto harmonize the current actual, desired and official recorded programexecution capacities for the selected computing node group, such as toadjust the current official capacity to the current desired capacity.The routine then continues to block 535 to designate the current desiredprogram execution capacity that will be provided by the modifiedselected computing node group as the updated current official recordedprogram execution capacity for the selected computing node group, andfurther initiates the program execution capacity modificationsdetermined in block 530. After block 535, the routine continues to block599 and returns.

FIG. 6 is a flow diagram of an example embodiment of a CapacityModification Attribution routine 600. The routine may be provided by,for example, execution of the PESSM modules 110 and 180 of FIGS. 1A and1B, respectively, and/or the PESSM module 340 of FIG. 3, such as may beinitiated from block 475 of routine 400 in FIG. 4.

In the illustrated embodiment, the routine begins in block 605, where itobtains an indication of the currently selected computing node group forwhich capacity modification attribution activities are to be performed.In block 625, the routine then identifies information about programexecution capacity availability changes that have occurred since a priortime, such as the prior harmonization activities that were performed forthe selected computing node group. The identifying of the changeinformation may include, for example, retrieving information that waspreviously stored by a PES Capacity Maintenance Manager module as partof monitoring the selected computing node group, although in otherembodiments the routine 600 may dynamically determine the information(e.g., by dynamically requesting a PES Capacity Maintenance Managermodule or other monitoring source to provide that information).

After block 625, the routine continues to block 630 to determine whetherany such capacity availability changes have occurred, and if notcontinues to block 699. Otherwise, the routine continues to block 635 toidentify information about program execution capacity modificationrequest events that have occurred since a prior time, such as the priorharmonization activities that were performed for the selected computingnode group. The identifying of the event information may include, forexample, retrieving information that was previously stored by a PESSystem Manager module as part of providing functionality for theselected computing node group, such as discussed in greater detail withrespect to FIGS. 4 and 5. In block 640, the routine then determines anyof the identified events that directly cause corresponding availabilitychanges, such as based on the types of events (e.g., automated systemoperations to replace unavailable computing nodes, received userinstructions that specify an immediate corresponding response, etc.).

After block 640, the routine continues to block 645 to select the nextavailability change identified in block 625, beginning with the first.In block 650, the routine then determines whether this change isdirectly attributable to one of the individual events determined inblock 640, and if so records that event as the cause for the selectedcapacity availability change. Otherwise, the routine in block 650attributes the cause for the selected capacity availability change asbeing a combination of the other identified events that were notdetermined in block 640. In block 660, the routine then determineswhether there are more capacity availability changes, and if so returnsto block 645 to select the next such capacity availability change.Otherwise, the routine continues to block 699 and returns.

FIG. 7 is a flow diagram of an example embodiment of a Program ExecutionService Capacity Maintenance Manager routine 700. The routine may beprovided by, for example, execution of the PESCMM modules 115 and 160 ofFIGS. 1A and 1B, respectively, and/or the PESCMM module 345 of FIG. 3,such as to assist in monitoring groups of computing nodes for users,including to determine actual program execution capacity that isavailable from computing node groups. In this illustrated embodiment,the routine 700 operates in conjunction with a program execution servicethat provides program execution capacity for executing programs onbehalf of multiple users, although in other embodiments some or all ofthe functionality of the routine 700 may be provided in other manners.

In the illustrated embodiment, the routine begins at block 705, where anindication is received to initiate gathering of information aboutcomputing nodes of one or more computing node groups, such ascontinuously or otherwise in a repetitive manner—while not illustratedhere, in some embodiments other modules and/or routines may dynamicallyrequest the routine 700 to generate and provide particular informationof interest, such as with respect to a particular computing node groupof interest. In block 705, the routine gathers performancecharacteristics information for one or more such computing nodes,including indications of the computing node groups to which thosecomputing nodes belong, for later use in generating aggregateinformation for the computing node groups. The information may begathered in various manners, including by pulling the information byrequesting particular computing nodes or associated modules (e.g.,associated VM manager components for virtual machine computing nodes) toprovide the information, by such computing nodes and/or associatedcomponents pushing the information to the routine 700 (e.g.,periodically; upon certain types of events, such as a detected errorcondition that will cause the computing node to shutdown or otherwisebecome unavailable; etc.), by monitoring network traffic and/orindividual resource usage by particular computing nodes; etc.

The routine then continues to block 710 to determine aggregateperformance characteristic information for one or more selectedcomputing node groups and to store that determined information for lateruse, such as for all computing node groups, computing node groups forwhich individual computing node information was just gathered, computingnode groups for which aggregate performance characteristic informationhas been requested or has not been generated recently, computing nodegroups for which individual computing node information is available foreach of the computing nodes in the computing node group, etc. It will beappreciated that aggregate performance characteristic information may begenerated in various manners, including when only partial information isavailable regarding the computing nodes of the group, such as beextrapolating or otherwise estimating individual performancecharacteristics information that is not available. In addition,particular performance characteristics information that is gatheredand/or aggregated may vary in various manners, such as to always collectcertain types of information for all computing node groups in certainembodiments, to collect certain types of information based on thecriteria specified for determined triggers for particular computing nodegroups, etc. Furthermore, while blocks 705-785 are illustrated in thisexample as being performed in a sequential manner, it will beappreciated that various blocks may instead be performed in othermanners in some embodiments. For example, in some embodiments, theinformation gathering activities of block 705 may be performed on acontinuous basis or near-continuous basis, but the aggregate informationgeneration of block 710 and/or other blocks may be performed onlyperiodically.

After block 710, the routine continues to block 715 to determine andstore current actual program execution capacity information for each ofone or more computing node groups, such as to reflect a current quantityof computing nodes of a group that are currently available, and/or oneor more other measures of program execution capacity for a computingnode group, and stores the determined information for later use. Whilenot illustrated here, in some embodiments and situations, the routinemay further immediately update a corresponding official recorded programexecution capacity for a computing node group to reflect the determinedcurrent actual program execution capacity for the computing node group,while in other embodiments will wait until a next corresponding group ofharmonization activities to update the official recorded programexecution capacity.

In block 720, the routine then optionally determines any capacitymodification triggers for any computing node groups that have beensatisfied for any computing node groups (e.g., based on performancecharacteristics information gathered in block 705 and/or aggregated inblock 710, and/or based on actual program execution capacity informationdetermined in block 715), although in other embodiments such triggersatisfaction determination may instead be performed at the end of anaggregation period of time, such as with respect to correspondingharmonization activities that are discussed in greater detail withrespect to FIG. 5. If any triggers are determined to be satisfied,information about such satisfied triggers is then stored for later use.In some embodiments and situations, satisfaction of a particular triggermay further initiate an immediate program execution capacitymodification for an associated computing node group, and if so such aprogram execution capacity modification activity may be initiated, andthe routine may further record causation information that links thatsatisfied trigger with that program execution capacity modificationactivity, such as for later use with respect to routine 600 of FIG. 6.

In a similar manner, the routine in block 725 determines whether anycomputing nodes of any computing node groups have been initiated to bemade available for the computing node group, but the initialization hasfailed or otherwise not completed within a specified period of time(e.g., 10 minutes). If so, the illustrated embodiment of the routineinitiates the immediate providing of a replacement computing node forany such computing nodes, and the routine may further record causationinformation that links that initialized computing node unavailability asthe cause of the initiated replacement activity. In other embodiments,such replacement activities may instead not be performed in an immediatemanner (e.g., instead may be aggregated along with other requests tomodify program execution capacity), and/or may instead be performed bythe routine 400 of FIG. 4. In addition, as part of initiatingreplacement activities for such unavailable computing nodes, the routinemay further take actions to terminate the unavailable computing node(e.g., if it is still running but is unresponsive).

In a similar manner, the routine in block 730 optionally initiates theimmediate providing of a replacement computing node for any computingnodes of computing node groups that were previously in use as part ofthe computing node groups but are now identified as having failed orotherwise become unavailable, and if so the routine may further recordcausation information that links that computing node unavailability asthe cause of the initiated replacement activity. In other embodiments,such replacement activities may instead not be performed in an immediatemanner (e.g., instead may be aggregated along with other requests tomodify program execution capacity) and/or may be performed by theroutine 400 of FIG. 4, and/or other types of automated determinationsmay be performed that can automatically initiate immediate changes toprogram execution capacity for particular computing node groups. Inaddition, as part of initiating replacement activities for anunavailable computing node, the routine may further take actions toterminate the unavailable computing node (e.g., if it is still runningbut is unresponsive). Furthermore, as discussed in greater detailelsewhere, the routine may in some embodiments consider other factorswhen determining whether to immediately perform the replacementactivities with respect to blocks 725 and/or 730, such as periods oftime for which an associated user will be or has been charged, an amountof time until a next harmonization activity is scheduled to be performedthat may affect the desirability of performing the replacementactivities, etc.

After block 730, the routine continues to block 785 to optionallyperform any housekeeping operations, including to update storedinformation as appropriate. After block 785, the routine continues toblock 795 to determine whether to continue, such as until an explicitindication to terminate is received. If it is determined to continue,the routine returns to block 705, and otherwise continues to block 799and ends. While the various activities described with respect to routine700 are illustrated in this embodiment as being performed by a differentmodule than the module that performs routine 400, in other embodimentssome or all of the functionality of the two routines may be performed bya single module or otherwise combined.

It will be appreciated that in some embodiments the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some embodiments illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel)and/or in a particular order, those skilled in the art will appreciatethat in other embodiments the operations may be performed in otherorders and in other manners. Those skilled in the art will alsoappreciate that the data structures discussed above may be structured indifferent manners, such as by having a single data structure split intomultiple data structures or by having multiple data structuresconsolidated into a single data structure. Similarly, in someembodiments illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. Accordingly, the invention is not limited exceptas by the appended claims and the elements recited therein. In addition,while certain aspects of the invention are presented below in certainclaim forms, the inventors contemplate the various aspects of theinvention in any available claim form. For example, while only someaspects of the invention may currently be recited as being embodied in acomputer-readable medium, other aspects may likewise be so embodied.

1-29. (canceled)
 30. A non-transitory computer-readable medium havingstored contents that, when executed, cause a computing system to:identify, by the computing system, and for a group of multiple computingnodes that execute one or more software programs between a first timeand a later second time and that have a change in an amount of programexecution capacity being provided by the group between the first timeand the later second time, a plurality of events that occur between thefirst time and the later second time and that each is capable ofresulting in a modification of program execution capacity being providedby the group; determine, by the computing system, multiple events of theplurality of events that are associated with the change in the amount ofprogram execution capacity; and provide, by the computing system,information about the determined multiple events as being associatedwith the change in the amount of program execution capacity.
 31. Thenon-transitory computer-readable medium of claim 30 wherein the storedcontents include software instructions that, when executed, furthercause the computing system to store information about one or morecapacity modification triggers that each has one or more criteria foruse in later initiating automated modifications to program executioncapacity of the group if the one or more criteria are satisfied, andwherein at least one of the determined multiple events corresponds tosatisfaction of at least one of the capacity modification triggersbetween the first time and the later second time.
 32. The non-transitorycomputer-readable medium of claim 31 wherein the at least one capacitymodification trigger further includes an indicated type of programexecution capacity modification to implement if the at least onecapacity modification trigger is satisfied, wherein the stored contentsfurther cause the computing system to determine that the one or morecriteria for the at least one capacity modification trigger aresatisfied and to implement the indicated type of program executioncapacity modification for the group, and wherein the change in theamount of program execution capacity is based at least in part on theimplemented indicated type of program execution capacity modification.33. The non-transitory computer-readable medium of claim 31 wherein thecomputing system is part of a program execution service that providesthe group of multiple computing nodes for a user of the programexecution service, wherein the stored contents further cause thecomputing system to receive information from the user to specify the atleast one capacity modification trigger, and wherein the storing of theinformation about the one or more capacity modification triggers isbased at least in part on the received information from the user. 34.The non-transitory computer-readable medium of claim 33 wherein thereceiving of the information from the user includes receiving, via agraphical user interface provided by the program execution service,information from the user for one or more parameter values to use withthe at least one capacity modification trigger and instructions from theuser to provide the group for use by the user, and wherein the storedcontents further cause the computing system to, in response to theinstructions, select the multiple computing nodes for the group from aplurality of computing nodes provided by the program execution service,and provide the selected multiple computing nodes for use by the user.35. The non-transitory computer-readable medium of claim 33 wherein thereceiving of the information from the user includes receiving, via oneor more programmatic interfaces provided by the program executionservice, instructions from a computing device of the user to provide thegroup for use by the user and to implement the at least one capacitymodification trigger for use with the group, and wherein the storedcontents further cause the computing system to, in response to theinstructions, provision the multiple computing nodes for use in thegroup, and provide the provisioned multiple computing nodes for use bythe user.
 36. The non-transitory computer-readable medium of claim 31wherein the one or more capacity modification triggers include multiplecapacity modification triggers that are each associated with an amountof program execution capacity modification, wherein the determinedmultiple events correspond to satisfaction of two or more of themultiple capacity modification triggers, and wherein the change in theamount of program execution capacity is based at least in part on acombination of the amounts of program execution capacity modificationassociated with the two or more capacity modification triggers.
 37. Thenon-transitory computer-readable medium of claim 31 wherein thecomputing system is part of a program execution service that providesthe group of multiple computing nodes for a user of the programexecution service, wherein the stored contents further cause thecomputing system to receive, from the user and after the first timewhile the group is executing the one or more software programs,instructions to modify the program execution capacity being provided bythe group, and wherein an additional one of the determined multipleevents corresponds to implementation by the program execution service ofthe instructions.
 38. The non-transitory computer-readable medium ofclaim 30 wherein the change in the amount of program execution capacitybeing provided by the group includes a change in quantity of computingnodes in the group, and wherein the providing of the information aboutthe determined multiple events includes identifying the determinedmultiple events together as a cause of the change in the amount ofprogram execution capacity.
 39. The non-transitory computer-readablemedium of claim 38 wherein the identifying of the determined multipleevents as the cause of the change in the amount of program executioncapacity includes identifying a combination of the determined multipleevents as the cause.
 40. The non-transitory computer-readable medium ofclaim 38 wherein the determining of the multiple events includesselecting all of the plurality of events as the determined multipleevents.
 41. The non-transitory computer-readable medium of claim 38wherein the determining of the multiple events includes selecting asubset of only some of the plurality of events as the determinedmultiple events.
 42. A computer-implemented method, comprising:providing, by one or more configured computing systems that provide aprogram execution service, a group of multiple computing nodes thatprovides program execution capacity at a first time for executing one ormore software programs for a user of the program execution service;identifying, by the one or more configured computing systems, aplurality of events that occur between the first time and a later secondtime while the group is executing the one or more software programs andthat each is capable of resulting in a modification of the programexecution capacity being provided by the group; determining, by the oneor more configured computing systems, multiple events of the pluralityof events that are associated with a change in an amount of programexecution capacity being provided by the group between the first timeand the later second time; and providing, by the one or more configuredcomputing systems, information identifying a combination of thedetermined multiple events as a cause of the change in the amount ofprogram execution capacity being provided by the group.
 43. Thecomputer-implemented method of claim 42 wherein the determined multipleevents include a capacity modification request specified by the userafter the first time for an indicated modification to the programexecution capacity being provided by the group, and wherein thedetermined multiple events include implementation of the capacitymodification request.
 44. The computer-implemented method of claim 42further comprising receiving configuration information from the userbefore the first time that specifies one or more capacity modificationtriggers for use in later initiating automated modifications to theprogram execution capacity provided by the group if one or morespecified criteria are satisfied, and wherein the determined multipleevents include at least one modification to the program executioncapacity from at least one of the capacity modification triggers beingdetermined to be satisfied.
 45. The computer-implemented method of claim42 wherein the program execution capacity of the group is measured basedon a quantity of computing nodes that are part of the group, and whereinthe change in the amount of program execution capacity being provided bythe group includes one or more computing nodes of the group becomingunavailable or one or more additional computing nodes being added to thegroup.
 46. The computer-implemented method of claim 42 wherein theprogram execution service provides multiple physical computing systemsthat each hosts one or more virtual machines for use by users of theprogram execution service, and wherein the providing of the group ofmultiple computing nodes includes selecting, by the one or moreconfigured computing systems, one or more virtual machines hosted on oneor more of the multiple physical computing systems to include in thegroup, and provisioning the selected one or more virtual machines foruse by the user.
 47. A system comprising: one or more hardwareprocessors of one or more computing systems; and one or more memorieswith stored instructions that, when executed by at least one of the oneor more hardware processors, cause the at least one hardware processorto: execute one or more software programs for a user on a group ofmultiple computing nodes; identify a plurality of events that occurbetween a first time and a later second time while the group isexecuting the one or more software programs and that are associated witha change in an amount of program execution capacity being provided bythe group between the first time and the later second time; and provideinformation about multiple events of the plurality of events that are acause of the change in the amount of program execution capacity.
 48. Thesystem of claim 47 wherein the stored instructions further cause the atleast one processor to determine, before the providing of theinformation, the multiple events based on the multiple events beingassociated together as a combination with the change in the amount ofprogram execution capacity.
 49. The system of claim 47 wherein thestored instructions further cause the at least one processor to:receive, before the executing of the one or more software programs,information from the user that requests the multiple computing nodes beprovided for the group and that specifies one or more capacitymodification triggers for use in later initiating automatedmodifications to the program execution capacity provided by the group ifone or more specified criteria are satisfied; provide the multiplecomputing nodes for the user; determine that the one or more specifiedcriteria for at least one of the capacity modification triggers issatisfied during the executing of the one or more software programs; andimplement, based on the determining, one or more automated modificationsto the program execution capacity provided by the group, and wherein themultiple events include the implementing of the one or more automatedmodifications.